Measuring and Driving AI Adoption
Every company has an AI strategy. Few have an AI measurement strategy. This guide covers both: three methods to measure adoption, and four levers to drive it through your existing management infrastructure.
Every company now has an AI strategy. Few have an AI measurement strategy. Even fewer have a plan for actually driving adoption once they can see where they stand.
Leaders across industries are investing heavily in AI tools, from code-generation assistants for engineering teams to AI writing tools for marketing, AI-powered CRMs for sales, and intelligent automation for operations. Yet when asked how much these tools are actually being used, or whether they are delivering ROI, most organizations cannot answer with confidence.
This guide addresses both sides of the problem. Part one covers how to measure AI adoption using three complementary methods. Part two covers how to drive adoption through recognition, coaching, performance management, and accountability.
The core argument is straightforward: measurement and improvement are two sides of the same coin. Organizations that build both into their existing management infrastructure will pull ahead. Those that treat AI adoption as a one-time rollout will fall behind.
Part One: Measuring AI Adoption
AI adoption is uniquely difficult to measure. It spans every function, usage does not equal value, AI assistance is often embedded invisibly within existing tools, and output quality varies widely. No single metric tells the full story.
Organizations need a multi-method approach that combines three types of signal: telemetry, experience sampling, and periodic surveys. Each captures something the others miss.
1. Telemetry Metrics
What it is: Collecting quantitative usage data directly from AI tools and the platforms they integrate with.
How it works: Organizations aggregate data from AI tool APIs, SSO providers, and workplace platforms to build dashboards showing who is using which tools, how often, and in what patterns.
Examples of telemetry metrics by function:
| Function | Metric Examples |
|---|---|
| Engineering | AI code suggestions accepted, AI-assisted PRs merged, Cursor/Copilot session frequency |
| Sales | AI-generated email drafts sent, AI CRM enrichment actions, AI call summary usage |
| Marketing | AI content drafts generated, AI image generation sessions, AI-assisted campaign builds |
| Customer Success | AI response suggestions accepted, AI ticket summaries viewed |
| Finance | AI-assisted report generation, AI anomaly detection reviews |
| Operations | AI automation runs triggered, AI document processing volume |
| HR | AI-assisted review drafts generated, AI sourcing tool queries |
Strengths: Objective and continuous, scalable, easy to track trends, can be broken down by team/role/geography.
Limitations: Measures activity not outcomes, cannot capture invisible AI use, tool fragmentation makes aggregation difficult, privacy concerns may limit granularity.
What to look for in a solution: The biggest challenge is fragmentation. Leaders need a centralized system that pulls usage data from all tools into a single view alongside other work signals (PRs merged, tickets closed, deals progressed) so AI adoption is visible in context, not in isolation.
2. Experience Sampling and Time-in-Motion Studies
What it is: Structured, periodic check-ins that ask employees to report on their AI usage in context.
How it works: Organizations use short, frequent prompts delivered via Slack to ask employees about recent AI usage. These micro-surveys capture perceived usefulness, time saved, barriers encountered, and workflow context.
Example experience sampling questions:
- “In the last 24 hours, did you use an AI tool to help with a work task?”
- “How much time do you estimate AI saved you this week?”
- “What was the most valuable AI-assisted task you completed recently?”
- “Did you encounter a situation where you wanted to use AI but couldn’t? What stopped you?”
Strengths: Captures context and perceived value, identifies barriers, works across all tools including unofficial ones, lightweight enough for recurring deployment.
Limitations: Subject to recall bias, requires consistent participation, can create survey fatigue if overdone.
What to look for in a solution: The key is timing and delivery. Check-ins should reach people at the right moment, delivered where they already work (i.e. Slack via a tool like Windmill). Two to three questions, weekly or biweekly, rotating across dimensions. Responses should flow into the same system where managers already look at team data.
3. Periodic Surveys
What it is: Lightweight, recurring surveys that assess AI adoption attitudes, proficiency, and perceived impact.
Example survey dimensions:
| Dimension | Sample Question |
|---|---|
| Tool Awareness | ”Which AI tools are you aware your organization provides?” |
| Proficiency | ”How confident are you in using AI tools effectively?” |
| Frequency | ”How often do you use AI tools in your daily work?” |
| Impact | ”To what extent have AI tools improved your work?” |
| Barriers | ”What prevents you from using AI tools more?” |
| Quality Concerns | ”How often do you need to significantly revise AI output?” |
What to look for in a solution: Traditional surveys are too heavy. Short, frequent pulses (three to five questions every week or two, delivered in Slack) get high participation and fresh data. Include an open-ended question like “What AI use case is working best?” and feature the responses at all-hands.
Comparing the Three Measurement Approaches
| Telemetry | Experience Sampling | Periodic Surveys | |
|---|---|---|---|
| Best for | Tracking adoption trends, license ROI | Usage patterns, barriers, time savings | Readiness, sentiment, training gaps |
| Not good for | Understanding why tools are/aren’t used | Scaling without automation | Detecting rapid changes |
| Key challenge | Tool fragmentation | Survey fatigue if poorly designed | Low response rates |
| Frequency | Continuous | Weekly or biweekly | Every two to four weeks |
| Effort | High initially, low ongoing | Medium | Low |
Part Two: Driving AI Adoption
Measurement without action is just monitoring. Driving adoption requires embedding it into existing management infrastructure. The four levers are recognition, one-on-one coaching, performance evaluation, and organizational accountability.
1. Recognition and Visibility
What it is: Creating visible, recurring moments where AI-powered wins are celebrated across the organization.
Why it works: Recognition rewards adopters (reinforcing behavior) and exposes everyone else to concrete examples. Most employees are not resistant to AI. They just do not know where to start. Seeing a colleague describe how they used Claude to cut a two-hour task to twenty minutes is more compelling than any training webinar.
Example recognition formats by function:
| Function | Example Shout-out |
|---|---|
| Engineering | ”Used Cursor to refactor our API error handling. 3-hour task took 45 minutes, caught 2 edge cases I would have missed.” |
| Sales | ”Fed our last 10 closed-won call transcripts into Claude. Found 3 objection patterns we hadn’t trained on.” |
| Marketing | ”Used ChatGPT for 15 email subject line variations. A/B tested top 5 and beat previous open rate by 12%.” |
| CS | ”Built a prompt that summarizes health signals before renewal calls. Prep: 30 min to 5 min per account.” |
| HR | ”Used AI to draft interview questions tailored to each resume. Interviewers said questions were noticeably better.” |
| Finance | ”Automated monthly variance analysis narrative. Report drafting: full day to two hours of review.” |
Strengths: Creates self-reinforcing loop, builds searchable use case library, costs almost nothing.
Limitations: Dies without leadership participation. Someone needs to seed the channel for the first weeks.
What to look for in a solution: The recognition tool should be embedded where people already communicate. It should support structured shout-outs that can be tagged, searched, and aggregated. Recognition data should flow into the same system used for performance management so managers can see AI shout-outs during reviews.
2. Manager One-on-Ones
What it is: Building AI adoption into the weekly manager-employee conversation as a standing agenda item.
Why it works: The 1:1 is where priorities, blockers, and development get discussed. When managers ask about AI every single week, it signals a real priority. Most underutilizers are not resistant, they are stuck: wrong tool, bad first experience, or unsure if they are “allowed.” The 1:1 is where a manager unblocks them.
Example one-on-one discussion prompts:
| Category | Prompt |
|---|---|
| Usage Check-in | ”What AI tools did you use this past week? Walk me through one example.” |
| Value Assessment | ”Did AI save you meaningful time this week? Where did it help most?” |
| Barrier Identification | ”Was there a task where you thought about using AI but didn’t? What stopped you?” |
| Skill Development | ”Is there an AI tool or technique you want to learn but haven’t explored?” |
| Quality Reflection | ”Did you run into quality issues with AI output? How did you handle it?” |
| Knowledge Sharing | ”Did you discover anything that could help others on the team?” |
What good coaching looks like: A manager notices from past 1:1 notes that their sales rep only uses AI for email drafts. They ask what else feels repetitive. The rep mentions 30 minutes of call prep. The manager suggests trying AI to summarize CRM notes. Next week the rep reports it cut prep time in half. The manager encourages them to share it in the recognition channel. Small moment, but multiplied across every manager and report, every week, it compounds into transformation.
Strengths: Personalized coaching, identifies invisible barriers, builds longitudinal record, naturally generates knowledge sharing.
Limitations: Entirely dependent on manager consistency. Without structured templates, defaults to whatever the manager remembers.
What to look for in a solution: The 1:1 tool needs structured, repeatable templates with AI adoption prompts built in. Managers should easily reference previous conversations to track trajectory. Data from 1:1s should feed into the broader picture alongside telemetry and survey responses.
3. Performance Reviews
What it is: Including AI adoption as a formal evaluation criterion, with a dedicated question in both the self-review and the manager evaluation.
Why it works: Performance reviews are the single most powerful signal for communicating what an organization values. When AI adoption is a rated dimension alongside job performance, collaboration, and leadership, it tells every employee: this is part of how we define excellence here.
Example self-review prompts:
| Dimension | Prompt |
|---|---|
| Impact | ”Describe 2-3 examples where AI meaningfully improved the quality, speed, or outcome of your work.” |
| Growth | ”What new AI tools or techniques did you learn? What do you want to explore next?” |
| Knowledge Sharing | ”How have you helped others on your team adopt or improve their use of AI?” |
| Quality Awareness | ”Describe a situation where you identified an issue with AI output and how you addressed it.” |
Example manager evaluation prompts:
| Dimension | Prompt |
|---|---|
| Adoption Level | ”Rate this person’s AI adoption relative to peers in a similar role.” (Leading / On track / Developing / Not yet started) |
| Effectiveness | ”Is this person using AI to produce genuinely better or faster work, or is usage superficial?” |
| Quality Judgment | ”Does this person demonstrate good judgment about when to rely on AI and when to apply manual rigor?” |
| Team Contribution | ”Has this person contributed to broader AI adoption through sharing, documentation, or mentoring?” |
| Development Areas | ”What AI skills or tools should this person focus on next?” |
The data problem: The biggest failure mode is the blank page problem. A manager stares at an empty text box and writes something generic. The solution: bring data into the review. The manager should have in front of them: telemetry showing tool usage trends, pulse responses describing how AI helped, recognition given and received, and notes from 1:1 conversations. When self-reported reflections sit alongside actual data, the conversation becomes substantive and the rating defensible.
Strengths: Clearest signal about organizational priorities, creates formal record for promotion/compensation, forces serious reflection from both sides.
Limitations: Only happens once or twice per year (lagging indicator). Without supporting data, becomes box-checking. Requires organizational commitment.
What to look for in a solution: The review platform should support custom questions added to any cycle. More importantly, it should pull in contextual data: usage stats, pulse responses, recognition, and 1:1 notes alongside the review form. It should also support AI-assisted writing that helps managers draft specific, evidence-based evaluations. Resulting ratings should be aggregatable so leadership can see AI proficiency across teams and the organization.
4. Accountability Mechanisms
What it is: Making AI adoption visible at the team and organizational level so leaders can identify and address gaps systematically.
Why it works: Recognition, 1:1s, and reviews all operate at the individual level. Organizations also need a macro view: where are pockets of low adoption, which teams have figured it out, which functions need more support, and is the company making progress quarter over quarter?
Team-level accountability:
| Mechanism | How It Works |
|---|---|
| Adoption dashboards | Aggregate telemetry, pulse responses, and recognition into team-level views. Updated weekly. |
| Manager reporting | Each manager reviews their team’s AI data in their own 1:1 with their manager. Creates upward pressure. |
| Team goals | Set explicit targets (e.g., “80% of team uses AI tools weekly by end of Q2”) alongside traditional targets. |
Function-level accountability:
| Mechanism | How It Works |
|---|---|
| Quarterly reviews | Each VP presents their function’s AI adoption trends, wins, and gaps. |
| Cross-functional benchmarking | Compare adoption rates across functions. If engineering is at 90% and finance at 30%, that gap needs attention. |
| Resource allocation | Lower-adoption functions may need more tools, training, or AI champions. Quarterly review is where these decisions get made. |
Company-level accountability:
| Mechanism | How It Works |
|---|---|
| Executive scorecard | Include AI adoption metrics alongside revenue, NPS, and other strategic KPIs. |
| Compensation alignment | Tie AI adoption progress to bonus structures at the leadership level. |
| Board reporting | Include AI adoption metrics in board updates for strategic priority. |
Strengths: Prevents AI adoption from becoming “talked about but not tracked,” identifies systemic gaps, enables data-driven resource allocation.
Limitations: Can feel surveillance-oriented without enablement. Team-level metrics can obscure individual variation.
What to watch for: Accountability without support backfires. Every accountability mechanism should be paired with enablement: training, tool access, dedicated learning time, and AI champions.
What to look for in a solution: The platform should aggregate telemetry, pulses, recognition, 1:1 notes, and review data into dashboards at team, function, and company level. Leaders should see trends, compare teams, and drill into context, not just “usage went up 20%” but why it went up and what’s driving it.
Comparing the Four Driving Mechanisms
| Recognition | One-on-Ones | Performance Reviews | Accountability | |
|---|---|---|---|---|
| Primary lever | Culture and peer influence | Individual coaching | Formal evaluation and career impact | Organizational visibility |
| Frequency | Continuous | Weekly | Quarterly or semi-annually | Monthly dashboards, quarterly reviews |
| Operates at | Individual and team | Individual | Individual | Team, function, and company |
| Best for | Normalizing AI use, spreading practices | Unblocking individuals, personalized coaching | Signaling priorities, informing compensation | Identifying systemic gaps, resource allocation |
| Key risk | Dies without leadership participation | Inconsistent without structured templates | Hollow without supporting data | Feels like surveillance without enablement |
| Effort | Low | Medium | Medium | High |
The four mechanisms work as a system. Recognition creates the culture. One-on-ones provide the coaching. Performance reviews formalize the expectation. Accountability ensures progress is tracked and resources allocated. Organizations that implement all four see compounding results.
Quality Control: The AI Slop Problem
As adoption increases, organizations face a decline in output quality driven by over-reliance on AI-generated content.
What AI Slop Looks Like
| Function | AI Slop Risk |
|---|---|
| Engineering | Code that passes tests but introduces subtle bugs or architectural debt |
| Sales | Generic outreach that fails to match company voice |
| Marketing | Content that is accurate but bland and derivative |
| Customer Success | Responses that miss nuance or escalation signals |
| HR | Review comments that are vague and repetitive across employees |
| Finance | Analyses with plausible but incorrect conclusions |
Why It’s Dangerous
AI slop is efficient to produce. More output looks productive by traditional metrics. Without quality checks, organizations optimize for volume at the expense of quality. This is insidious because it is hard to detect until damage accumulates.
Guardrails
1. Clear AI-use policies by function. Define where AI is encouraged, where it requires review, and where it is inappropriate.
2. Review built into workflows. AI output should be treated as a first draft. Build explicit review steps into AI-assisted workflows.
3. Quality metrics alongside adoption. Track quality indicators (response quality scores, code review rejection rates, content engagement, deal conversion rates, feedback specificity). If adoption goes up but quality goes down, intervene.
4. Training on prompting and critical review. Many quality issues stem from lack of training on effective prompting and evaluating AI output.
Stay in the loop
Get the latest updates, insights, and news from Windmill delivered to your inbox.