A tool demonstration produces a patch in minutes. A finance discussion asks what that means for the engineering budget. Between those two events sits most of the measurement problem: the patch may need review, its tests may omit a requirement, and the work it saves may not be the constraint limiting delivery.
An engineering leader needs an answer they can defend without overselling or dismissing the opportunity. The useful question is whether a particular approved form of assistance improves a particular workflow under the team's actual conditions. The answer may be positive, negative, mixed, or too uncertain to act on. Those are legitimate outcomes when the trial was designed to inform a decision.
This guide follows a hypothetical pilot for routine internal integration changes. The example includes invented tasks and proposed measurement rules, not reported company results. The output is a pilot measurement brief that distinguishes time, quality, capacity, and money. It also records conditions that would prevent a larger rollout.
The aim is to make the next decision smaller and more evidence-based. If assistance helps prepare test fixtures but adds review cost to unfamiliar integrations, you can retain one use and change the other. You do not need a single percentage that pretends to describe every engineer and every task.
A worked pilot model that includes the reviewer
The engineering calculator's fictional month starts with 200 preparation hours and 20 review/rework hours. Its AI-assisted scenario uses 140 preparation hours and 40 review/rework hours. Looking only at preparation would report 60 hours released. Including review reduces that modeled difference to 40 hours.
At the supplied internal cost of $150 per hour, those hours have a modeled capacity value of $6,000. Recurring tool cost of $2,000 and setup cost of $3,000 leave $1,000 after subtracting both costs from modeled capacity value for this period. There is no observed cash saving, revenue increase or real trial result in these numbers.
Decide what finance needs to see
Ask whether the decision concerns capacity, delivery, quality, spending or an actual cash expense. A team that delivers more accepted work with the same payroll has a different result from a team that reduces a cash outlay. Record the proposed use of released capacity instead of assigning it an automatic financial benefit.
GAO's estimating principles support explicit scope, assumptions and updates using actual costs. They do not validate AI ROI. Method reference.
Give the trial an interpretable design
The blank log records task category, eligible scope, condition, preparation, review, rework, acceptance, failure severity, missing observations and tool version. Agree the timing rule for concurrent work before collecting results. Keep unsuccessful tasks in the denominator where the design requires them, and explain missing data.
SPACE describes productivity across multiple dimensions. Author publication summary. METR's February 2026 update, already discussed in this guide, explains selection and timing limitations; do not borrow a study's headline percentage as the assumed benefit for your team.
Equal counts of quality failures also need interpretation. Severity and task scope can differ. The worksheet does not automatically recommend a staffing change or pronounce a pilot successful.
Choose a pilot question that can change a decision
Start with a workflow and a decision owner. The integration pilot asks whether approved assistance helps complete certain internal changes under the existing acceptance process. That answer could inform whether to retain the workflow, expand to a comparable task, or change the assignment. Asking whether AI makes engineering more productive in general is too broad for a local pilot to settle.
Write the possible next actions before the trial. A favorable result might justify another bounded use, not automatic expansion into sensitive repositories or production actions. A poor result might mean the context or task selection should change rather than that all assistance is useless. An uncertain result might require better timing or a different comparison.
Also identify the organizational constraint. If release approval is the bottleneck, faster code drafting may increase the queue without changing delivered value. If a particular fixture task consumes scarce specialist time, improving it may create useful capacity. The pilot should measure the part of the workflow the decision actually concerns. That keeps a demonstration from answering a budget or staffing question it was never designed to address.
A hypothetical pilot: routine integration changes with a common acceptance standard
An engineering group considers approved AI assistance for small internal integration changes. The proposed tasks include updating a field mapping, adding an error response, and preparing a test fixture. The leader wants to know whether the workflow reaches accepted changes with less total effort. They do not begin by promising a headcount reduction or a fixed delivery increase.
The pilot uses permitted nonproduction material and the normal review process. It distinguishes drafts, reviewed changes, merged changes, and work that remains unresolved. Before any outcomes are observed, the team writes the following measurement conditions.
The case is hypothetical; its acceptance conditions are proposed local decisions, not measured project results or industry benchmarks.
Completion definition
The initial report counts a task as complete when the assistant opens a pull request. The ordinary workflow counts completion after human review and required checks, so the two counts describe different stages. Inspect an agreed task-state definition and a sample review record.
The case accepts this part only when both paths count completion at the same reviewed acceptance stage. It rejects the proposal when draft pull requests are compared with accepted ordinary changes.
Next, write the completion rule and reclassify every pilot task against it. The review asks: What exactly must be true before this task enters the completed column? This requires measurement design. Count the effort spent agreeing the state definition and auditing task classifications in the experiment.
Task mix
The assisted sample contains simple field mappings while the comparison sample contains unfamiliar integration failures. A favorable difference could reflect assignment rather than assistance. Inspect a task inventory with complexity, familiarity, and dependency notes.
The case accepts this part only when task differences are recorded and the comparison remains appropriately bounded. It rejects the proposal when the report attributes task-mix differences entirely to the tool.
Next, group comparable tasks and disclose differences that cannot be matched. The review asks: Which task characteristics could explain this comparison without any tool effect? This requires experimental judgment. Count the effort spent classifying tasks and reviewing the limits of the comparison in the experiment.
Review and correction time
The pilot log records generation time but omits the senior reviewer's work explaining an incorrect mapping. That omission makes the new workflow appear cheaper by transferring effort to another person. Inspect a preparation, review, and correction log by task.
The case accepts this part only when the measurement includes the effort needed to reach the shared acceptance standard. It rejects the proposal when review work is excluded because it occurs after the first draft.
Next, ask participants to record review and correction separately from generation. The review asks: Whose work became invisible when we called this task finished? This requires workflow accounting. Count the effort spent collecting and interpreting review records in the experiment.
Quality and escaped issues
A change passes the pilot's initial checks but later requires a correction. Counting the earlier acceptance alone would hide rework that belongs to the delivery decision. Inspect a change-linked issue and rework record within the agreed observation window.
The case accepts this part only when the report separates accepted work from later corrections and unresolved observations. It rejects the proposal when known rework is omitted to preserve a favorable speed result.
Next, link follow-up issues to the original task and state the observation limit. The review asks: Which quality consequences were visible before the review date, and which were not? This requires delivery interpretation. Count the effort spent tracing follow-up issues and assigning rework effort in the experiment.
Concurrent agent work
An engineer runs an assistant while reviewing another task. Summing each assistant session duration as labor time or ignoring overlapping attention can both distort the effort estimate. Inspect a timing rule that separates elapsed time, active attention, and overlapping work.
The case accepts this part only when the report uses a stated timing convention and identifies uncertain overlap. It rejects the proposal when concurrent work is converted into a precise labor saving without a defensible rule.
Next, record the timing convention and flag sessions that cannot be attributed cleanly. The review asks: Are we measuring elapsed delivery time, active effort, or available capacity? This requires measurement discipline. Count the effort spent reconstructing overlapping work and reviewing timing uncertainty in the experiment.
Costs that change and costs that remain
The initial ROI estimate includes a subscription charge and assumes every minute saved becomes payroll savings. The team has not identified any staffing or spend change that would convert that time into cash. Inspect a cost model separating cash outlay, allocated cost, and proposed capacity use.
The case accepts this part only when cash savings are claimed only for a stated and supportable cost change. It rejects the proposal when time saved is presented as realized cash savings without a corresponding decision.
Next, show capacity and cash scenarios separately with their assumptions. The review asks: What actual budget line changes if this workflow takes less effort? This requires financial reasoning. Count the effort spent building the cost scenarios and reviewing finance assumptions in the experiment.
Data and acquisition work
The trial requires permission review, integration setup, and revised data handling. Those costs are not in the first estimate because they occur outside an engineer's coding session. Inspect an acquisition and permitted-data setup log.
The case accepts this part only when setup and recurring review costs are included or explicitly excluded with a reason. It rejects the proposal when the reported ROI ignores necessary governance or integration effort.
Next, ask the relevant owners to estimate and record their part of the pilot. The review asks: Which cost must recur when the task or tool changes? This requires cross-functional planning. Count the effort spent collecting setup effort and reviewing retention and access in the experiment.
Decision after an uncertain result
The leader expects one rollout answer, but some tasks may help while others remain inconclusive. Treating uncertainty as failure or rounding it into success would both lose useful information. Inspect a task-level result summary and a prewritten decision rule.
The case accepts this part only when the next action distinguishes retain, narrow, retest, and stop conditions. It rejects the proposal when an ambiguous pilot becomes a universal rollout or staffing claim.
Next, report task-level findings and choose the smallest justified next test. The review asks: What does this observation authorize us to try next, and what does it leave unanswered? This requires decision ownership. Count the effort spent preparing the decision brief and reviewing unresolved evidence in the experiment.
Separate task efficiency, delivery, and financial value
Task efficiency concerns effort or elapsed time for a defined task. Delivery concerns accepted work through the relevant process, with quality and operating consequences. Financial value concerns costs or benefits the organization can explain through a budget or business decision. These measures are related, but one does not automatically establish the next.
A shorter task may free time. Whether that time becomes a useful capacity change depends on the surrounding workflow and what the team does with it. Whether it becomes cash savings depends on a concrete spend or staffing decision. A report should not multiply minutes by a salary rate and call the result realized savings without stating that assumption.
Use a worksheet with distinct columns for direct cash outlay, allocated effort, potential capacity, and benefits requiring another decision. The distinction is practical rather than a prescribed accounting standard. Ask finance to review the organization's treatment. For the hypothetical pilot, a subscription charge is a cash cost, senior review effort consumes capacity, and a faster customer feature would require a separate value estimate. Keeping those categories visible makes a small trial useful without inventing a financial result.
Build a comparison that does not hide the task mix
A randomized assignment may help answer a causal question when it is feasible and properly designed, but many local trials use simpler comparisons. Be candid about the design. A before-and-after comparison can be affected by familiarity, changing requirements, staffing, or unrelated process improvements. A matched-task comparison can still leave differences you cannot fully measure.
Record task size, relevant familiarity, dependencies, risk, and the acceptance stage. Choose the descriptors because they may affect the result, not because they make the report look scientific. Avoid excluding difficult assisted tasks while retaining difficult ordinary tasks. Failed or abandoned assignments belong in the trial record when they are part of the decision being tested.
The internal integration example can compare similar field mappings, but an unfamiliar error-handling change may need its own category. If the samples are too small or different to support a stable estimate, report the observations and their limits. A useful decision brief can say that one assignment appeared promising while the broader effect remains unknown. That is more defensible than a precise percentage built from incomparable work.
Report the denominator as carefully as the numerator. State how many eligible tasks were offered, attempted, abandoned, and accepted. A completion count without the offered workload can hide selective participation. Preserve the reasons for exclusions so that the next reviewer can assess whether the sample describes the workflow being considered.
Count preparation, attention, review, and correction
Choose a timing convention before results arrive. Elapsed time and active effort answer different questions. An assistant may run while an engineer does other work, a reviewer may wait for an external decision, and several agents may overlap. The measurement should identify what is counted rather than pretending every duration is interchangeable.
METR's February 2026 update explains difficulties with participant and task selection and with concurrent-agent time measurement. Its earlier early-2025 slowdown concerned a specific experienced-developer setting; it is not a universal current effect. The later update does not supply a dependable general speedup either. Use these limits as reasons to state your timing and selection rules clearly.
Record review and correction even when the engineer feels that the first draft saved time. A task can be faster for its author and more expensive for the reviewer. Record setup and context preparation when they are necessary to repeat the workflow. If a trial contains uncertain timing, label that uncertainty. Do not repair the log by guessing a favorable value after the fact. A wider range or a qualitative observation can be more honest than unsupported precision.
Measure quality without treating a metric as the product
Decide which defects or rework matter to the pilot and how long you will observe them. Test passage, reviewer approval, and later operating outcomes each reveal something different. A green test suite does not establish that the feature meets every requirement, while a later defect may have causes unrelated to assistance. Link observations to the task before drawing conclusions.
The relevant quality checks depend on the workflow. A field mapping may need independently approved expected values. An authorization change may require negative-path checks. A dependency change may need acquisition and compatibility review. Include the conditions in the acceptance definition rather than measuring code volume and adding quality as an afterthought.
DORA's qualitative analysis of engineering responses describes generation alongside checking and integration tensions. It is observational, drawn from a particular respondent setting, and does not prove that adoption causes a given quality outcome. The pilot should use it to consider what work might be missed, not to borrow a causal result. Likewise, a completed-task finding from a field experiment does not establish maintainability, production reliability, or staffing needs in your organization.
Include security and acquisition costs without creating a compliance fiction
The pilot needs an approved data boundary, a known integration, and clear action permissions. Synthetic examples may be sufficient to test a preparation task. A broader trial involving internal code may require a separate review. Do not silently expand access because the approved sample cannot answer every question.
NIST SP 800-218 can inform the general secure development practices around a software change. NIST's generative AI risk profile discusses voluntary risk measurement and management. Neither certifies the tool or supplies the team's current error rate. This page does not present model-development guidance as a universal requirement for ordinary code assistance.
Ask the data, security, procurement, and integration owners what their work contributes to the trial. Record recurring review separately from initial setup where that distinction matters. A control may be worthwhile even when it reduces a headline ROI estimate; its purpose is not to make the estimate favorable. If the organization cannot establish acceptable terms or permissions, narrowing or stopping the pilot is a legitimate decision. The evidence brief should make that condition visible.
Use the evidence to discuss roles without inventing a staffing ratio
A pilot can show where work moved. An engineer may spend less time preparing boilerplate and more time checking requirements. A senior reviewer may need more capacity because drafting increased. A manager may need to redesign task assignment. Those observations concern responsibilities; they do not by themselves determine how many people the organization should employ.
Choose skill development from the trial. If the most expensive corrections involve domain mappings, improve domain knowledge and independent fixture design. If the team cannot interpret timing, improve measurement. If review is the constraint, practice defining smaller changes and clearer acceptance records. A generic recommendation to become an AI expert may miss the actual need.
Ask people what work they want to own next and what the organization needs. An engineer interested in deeper implementation may choose different practice from a manager interested in workload design. Preserve room to test those choices. Using a limited pilot as proof that a role is safe or doomed replaces a professional conversation with a claim the pilot cannot support. The evidence can inform that conversation while leaving the larger decision open.
A 30-day engineering AI measurement plan
In the first week, define the workflow, completion stage, timing convention, and data permissions. Recover a small comparison sample and write the decision rule. Make sure participants understand that failures and review effort belong in the record. A trial designed to reward favorable reporting will produce a weaker decision brief.
In the second week, run the bounded assignments and log preparation, attention, review, and corrections. In the third week, inspect task mix, follow-up issues, and missing observations. Ask finance to review the difference between capacity and cash assumptions. Do not broaden access or promise a rollout merely because early tasks look encouraging.
In the fourth week, prepare the result by task category and decide what to retain, revise, retest, or stop. State which conclusions remain uncertain. If the result suggests a useful narrower assignment, propose that next test with its own permissions and acceptance conditions. Keep a personal action separate, such as learning to design an independent fixture or discussing a review-capacity responsibility with your manager.
A pilot brief that preserves uncertainty for the next decision
The brief should give its reader a reason to choose the next action. It should not hide unfavorable tasks in an appendix or blend task speed, quality, and money into a single success score. The following hypothetical handoffs show how a measurement issue can remain inspectable after the pilot owner has moved on.
State the observation window and the completion rule beside the result. Report missing timing or unresolved rework as missing rather than zero. Keep participants' identities and sensitive material inside the approved access boundary. The decision-maker needs the conclusion and its support, not unnecessary copies of internal code or personal performance data.
Completion definition handoff. Attach an agreed task-state definition and a sample review record to the pilot measurement brief. Write the unresolved review question: What exactly must be true before this task enters the completed column? The next person should be able to perform this action: write the completion rule and reclassify every pilot task against it.
Review and correction time handoff. Attach a preparation, review, and correction log by task to the pilot measurement brief. Write the unresolved review question: Whose work became invisible when we called this task finished? The next person should be able to perform this action: ask participants to record review and correction separately from generation.
Concurrent agent work handoff. Attach a timing rule that separates elapsed time, active attention, and overlapping work to the pilot measurement brief. Write the unresolved review question: Are we measuring elapsed delivery time, active effort, or available capacity? The next person should be able to perform this action: record the timing convention and flag sessions that cannot be attributed cleanly.
Data and acquisition work handoff. Attach an acquisition and permitted-data setup log to the pilot measurement brief. Write the unresolved review question: Which cost must recur when the task or tool changes? The next person should be able to perform this action: ask the relevant owners to estimate and record their part of the pilot.
Compare your options
| Case issue | Evidence to retain | Acceptance condition | Reason to stop |
|---|---|---|---|
| Completion definition | an agreed task-state definition and a sample review record | both paths count completion at the same reviewed acceptance stage | draft pull requests are compared with accepted ordinary changes |
| Task mix | a task inventory with complexity, familiarity, and dependency notes | task differences are recorded and the comparison remains appropriately bounded | the report attributes task-mix differences entirely to the tool |
| Review and correction time | a preparation, review, and correction log by task | the measurement includes the effort needed to reach the shared acceptance standard | review work is excluded because it occurs after the first draft |
| Quality and escaped issues | a change-linked issue and rework record within the agreed observation window | the report separates accepted work from later corrections and unresolved observations | known rework is omitted to preserve a favorable speed result |
| Concurrent agent work | a timing rule that separates elapsed time, active attention, and overlapping work | the report uses a stated timing convention and identifies uncertain overlap | concurrent work is converted into a precise labor saving without a defensible rule |
| Costs that change and costs that remain | a cost model separating cash outlay, allocated cost, and proposed capacity use | cash savings are claimed only for a stated and supportable cost change | time saved is presented as realized cash savings without a corresponding decision |
| Data and acquisition work | an acquisition and permitted-data setup log | setup and recurring review costs are included or explicitly excluded with a reason | the reported ROI ignores necessary governance or integration effort |
| Decision after an uncertain result | a task-level result summary and a prewritten decision rule | the next action distinguishes retain, narrow, retest, and stop conditions | an ambiguous pilot becomes a universal rollout or staffing claim |
Write a pilot measurement brief before collecting a favorable story
Choose one workflow whose result could change a practical decision. Use this worksheet to separate observation from interpretation. It does not establish an accounting treatment, causal estimate, or staffing recommendation by itself.
A reflection tool for planning your next step, rather than a predictive score.
- Name the workflow, decision owner, and possible next actions.
- Define completion identically for assisted and comparison work.
- Record task characteristics that could explain differences.
- Choose elapsed-time and active-effort rules and label uncertain overlap.
- Include preparation, review, correction, setup, and follow-up costs.
- Separate cash changes from potential capacity and assumed business benefits.
- Report task-level findings, failures, missing observations, and limits.
- Choose the smallest justified next trial and a skill to practice.
The hypothetical pilot owner discovers that draft pull requests and accepted changes were being counted together. They correct the completion rule before interpreting the trial. They also separate reviewer effort from author drafting time so that a faster first draft cannot hide a transferred cost. No outcome is assumed in this example.
The leader notices a personal reluctance to report uncertainty because a confident percentage would make the budget presentation easier. They choose to present the bounded observations and name the next test instead. That is an action about their own decision behavior, not proof that the pilot will succeed.
If you know the evidence is incomplete but feel pressure to present certainty, examine how that pressure shapes your choices and actions. AI and the Andes Retreat is a five-day seminar in Peru's Sacred Valley for 8-15 participants, with no plant medicine. It offers space to examine the decisions, behavior, and actions shaping your life. You can bring a question about your professional direction and choose an action to test afterward. It is not a coding course or a promise about hiring, income, or career security.
Your working notes
Use this page while you think. Your notes stay in this browser tab; download them before you close it.
What does this question change for you?
A practical answer can change how you work. It can also leave a personal decision unresolved: what to learn, what to stop postponing, or which responsibility to take next. Name that decision before collecting another prediction.
Work through your next decision. Your notes stay in the browser tab, and the exercise gives you no predictive score.
If a recurring pattern keeps preventing action, AI and the Andes Retreat offers five days of practical inquiry in Peru's Sacred Valley, followed by support and accountability. Read about the seminar and assess whether that work fits your question.
Frequently asked questions
Search research and audience discussions informed these questions. Some wording has been adapted to the task covered by this guide.
Can generative AI improve developer productivity?
It can be useful in particular tasks, but measure the effect under your workflow rather than assume a universal gain. Define completion and count preparation, review, corrections, and follow-up work. The integration pilot asks about accepted changes, not generated patches. Published findings have task and participant limits; they do not establish a staffing ratio. A bounded result can justify retaining or retesting one assignment while leaving other tasks and broader business value unresolved. Sources: METR experiment update, developer field experiments.
How should I measure developer productivity with AI?
Define the accepted task and timing convention before collecting results. Include preparation, active attention, review, corrections, and necessary handoffs. Record task mix and quality observations instead of comparing draft volume with accepted ordinary work. For the integration pilot, both paths must reach the same completion stage. Label missing or overlapping timing honestly. The result should support a bounded workflow decision and disclose what it cannot establish about the organization as a whole.
How should I calculate ROI from developer productivity assistance?
Separate known cash outlay, allocated effort, potential capacity, and benefits that require another decision. Include setup, recurring review, integration, and correction costs. Do not turn every minute saved into realized payroll savings without identifying the budget change. For the integration pilot, show scenarios and uncertain assumptions rather than one invented return figure. Ask finance to review the organization's treatment and keep the workflow evidence separate from the commercial interpretation.
What productivity gain should engineering and finance expect an AI pilot to demonstrate?
Agree what the pilot should establish for the actual decision instead of assuming a universal gain. Measure preparation, review, rework, accepted outcomes, quality and complete costs for defined eligible tasks. Distinguish released capacity from cash savings. The synthetic calculator releases 40 modeled hours after review, but it does not establish that your team would achieve that result. Record the result that would justify extending, changing or stopping the trial.
How can I improve a software workflow with AI assistance?
Choose a task that addresses an actual constraint and test a limited assignment. Assistance may help prepare fixtures or summarize approved context while leaving review as the bottleneck. In the integration case, define the same acceptance stage and record where work moves. Retain a narrower use if it helps and revise tasks that create excessive correction. A general instruction to use more AI does not identify which part of delivery should improve or how you would know.
How should I evaluate productivity claims about AI agents?
Specify the actions and responsibility boundary before evaluating speed. An agent preparing a patch differs from one running checks or taking production actions. Count the human attention, review, corrections, and operating consequences required to reach the accepted task. Record overlapping work and timing uncertainty. For the integration pilot, greater action capability does not authorize broader access or prove a productivity gain. Test the particular permitted workflow rather than interpreting the agent label as a result.
How can I measure business impact beyond coding speed?
Connect the accepted workflow to a business decision and disclose the assumptions between them. A faster field mapping may create capacity without changing revenue, costs, or the delivery bottleneck. Identify which benefit would require another operational or budget choice and keep it separate from observed task effort. For the integration pilot, report quality, completion, and costs first. Do not assign an invented monetary value merely to make the ROI table complete.
Are generated lines of code a useful productivity measure?
They do not establish accepted delivery or business value by themselves. A larger patch can create more review and maintenance obligations, while a small change can resolve the relevant requirement. In the integration pilot, measure completion under the shared acceptance standard and include corrections. Keep code volume as a descriptive observation only if it serves a defined question. Do not treat it as a substitute for quality, operating ownership, or the work the organization intended to finish.
How should I respond when management assumes a fivefold AI speedup?
Ask which tasks, completion standard, and evidence support the assumption. Propose a bounded measurement plan rather than accepting or rejecting a universal multiplier. Include review, corrections, task mix, and operating work. The public post's fivefold claim is not a measured fact here. For the integration case, show what the pilot can establish and what remains unknown. Keep staffing commitments separate until the broader obligations and evidence have been reviewed by their accountable owners.
How should I assess a surprising AI productivity headline?
Read the original evidence before turning the headline into a decision. Check what was measured, who participated, how tasks were selected, and which comparison supports the finding. METR's early-2025 slowdown was bounded; its 2026 update describes selection and timing problems that prevent a dependable general current estimate. DORA's qualitative responses are observational. Use these limits to design your own task-specific pilot instead of treating a surprising public claim as a universal answer. Sources: METR experiment update, DORA qualitative analysis.
What costs belong in an AI-assisted engineering implementation estimate?
Include tool charges, approved integration work, context preparation, security review, training, evaluation, corrections, and recurring ownership for the defined workflow. Check current official prices and label unpriced obligations as unknown. In the hypothetical delivery trial, count reviewer attention as well as author effort, while keeping elapsed time separate. A subscription quote does not establish the cost of an accepted operating workflow. Compare the same task and time horizon before presenting the estimate as a decision input.
What should count as success in an AI-assisted engineering trial?
Set success conditions before inspecting the result: the accepted task, quality constraints, permitted data use, and a meaningful improvement measure. Include a condition that would stop or narrow adoption. In the delivery trial, faster drafting would fail the proposed acceptance rule if it transferred unaccepted correction work to reviewers. Retain individual failures and differences between tasks. A local successful trial supports the next scoped decision; it does not establish a staffing reduction or an organization-wide causal productivity effect.
Should story points be the main measure of AI-assisted engineering efficiency?
Use them only for a clearly defined local planning question, and do not assume a change in points establishes an accepted-work improvement. Check whether task mix, estimation conventions, and completion stages changed. In the integration pilot, a larger assigned estimate is not evidence that the assistant produced more value or reduced total effort. Retain actual acceptance, correction, and attention observations alongside any planning measure. Explain which comparisons the evidence supports instead of converting an estimate into a universal productivity unit.
How should I account for time spent verifying an AI-generated answer from a colleague?
Count the investigation needed before the answer becomes usable, including locating sources, checking consequential claims, and correcting unsupported statements. Ask the preparer to attach verified evidence and uncertainty at handoff. In the integration pilot, a generated explanation of a field mapping is not completed work if someone else must establish the actual contract from scratch. Record that effort under the relevant task rather than treating the fast reply as a saving or assuming every generated answer creates extra work.
How should I evaluate the ROI of upfront architecture preparation versus iteration?
Compare bounded alternatives over the same accepted task and time horizon, including preparation, corrections, and remaining operating obligations. Record what the design work is meant to resolve and which evidence would reject that assumption. For the integration pilot, inspect whether an independent interface record prevents repeated mapping investigations or simply adds unused documentation. Keep potential future benefits as scenarios until observed. This guide does not supply a benchmark return or adopt the source post's anecdotal costs.
How are AI capability performance and engineering productivity different measures?
A capability evaluation asks whether the output satisfies its defined task cases; a productivity evaluation asks what work is required to reach the accepted engineering outcome. Establish both scopes explicitly. In the integration pilot, a correct draft mapping can still require context preparation, permission review, and correction of surrounding code. Conversely, a fast response may fail the accepted contract. Do not substitute a model or tool score for workflow effort, quality, and operating evidence that the organization actually needs.
How should I compare another company's AI impact report with our engineering workflow?
Inspect the original report's task definitions, completion stage, participants, task mix, and cost boundary before comparing its figures. Check whether the report describes self-reported experience, observed activity, or an evaluated outcome. In the integration pilot, a count of drafts cannot be compared directly with another organization's accepted changes. State which differences remain unresolved. Use an external report to frame a local inquiry rather than treating its result as evidence of this team's gain or expected staffing needs.
How should I choose an AI tool for a developer-productivity trial?
Select a permitted task and the evidence needed to evaluate it before comparing tools. Check current official information about data use, configuration, action permissions, integration, and actual charges. This guide does not rank current products. For the integration pilot, choose a bounded assignment and an ordinary acceptance process that can reveal corrections and transferred effort. Compare the tool with a viable simpler workflow, and retain failures. A fast demonstration alone does not establish a useful productivity investment.
Does an interesting AI engineering experiment necessarily improve organizational productivity?
No. Define whether the work is learning, investigation, or delivery and identify the decision it supports. An experiment may be worthwhile without producing an immediate delivery improvement. In the integration pilot, exploring a new assistant is separate from finishing the approved mapping task. Set a bounded learning objective and report what was learned, including failure, instead of counting experimental output as accepted production work. Discuss the priority explicitly rather than assuming novelty either creates value or wastes time.
How should I use the percentage of engineering time spent on maintenance?
Treat it as a description of allocation under a stated denominator, not proof that the maintenance is valuable or excessive. Define what work is included and how overlapping attention is recorded. In the integration pilot, repeated mapping investigations and a planned repair may both use time but have different accepted outcomes. Pair the allocation with specific obligations and consequences. A changed percentage can reflect a different task mix or total workload, so do not interpret it automatically as an AI-related saving.
Sources and evidence
These guides were drafted with AI assistance and checked against the linked sources. Examples labeled hypothetical illustrate a decision; they are not participant stories or measured results.
Studies answer specific questions about particular people, tasks, and tools. Their dates and limits matter. A result about task assistance is not a forecast for an entire profession.
- NIST SP 800-218: Secure Software Development Framework
- NIST AI 600-1: Generative Artificial Intelligence Profile
- METR: We are Changing our Developer Productivity Experiment Design
- DORA: Balancing AI tensions: Moving from AI adoption to effective SDLC use
- The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers
- O*NET: Software Developers
- Observed audience question: How tech companies measure the impact of AI
- Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Program Costs (2020-03-12)
- The SPACE of Developer Productivity: There's more to it than you think (2021-02)
Where the audience questions came from
These linked discussions and comments supplied questions and wording. They do not establish the truth of the replies, verified professional identities, or how common a concern is.
- Observed audience question: How do you measure general business impact?
- Observed audience question: Nothing is based on real metrics it’s all “how many lines of code did this do?
- Observed audience question: Managers decided AI is worth 5x speedup; how do I explain to them how it really works?
- Observed audience question: The AI coding productivity data is in and it's not what anyone expected
- Observed audience question: Company is trying to increase engineering efficiency with AI(LLM)
- Observed audience question: The same goes when a question is asked and someone helpfully chimes in with “I have no clue but ChatGPT says…” Thanks, now I’m wasting my time fact checking a robot hallucination for you, how exactly is this helpful to me?
- Observed audience question: - Has anyone actually quantified the ROI of doing thorough upfront architecture work vs just iterating?
- Observed audience question: Do you feel like pursuing cool stuff to build may sometimes be counterproductive to progress from an organizational perspective?
- Observed audience question: This is why the only axis of control stakeholders should have into tech debt or "product quality" or whatever you want to call it is "what % of your time are you spending on this?
- Audience discussion informing the engineering question
