FSE 2023 Keynote

Towards AI-Driven Software Development: Challenges and Lessons from the Field

Eran Yahav

TL;DR — AI is poised to transform the entire software development lifecycle, not just code generation. The key challenges are: adapting the SDLC to autonomous agents, designing the right human-AI interaction, making AI contextually aware of your codebase, and establishing trust in AI suggestions. The fundamental insight: GenAI pays off when the cost of specifying intent plus consuming the result is much less than doing the work manually.

The Big Picture

AI is becoming powerful enough to change not just the level of abstraction in software development, but the nature of interaction between humans and machines. We are moving toward a world where every engineer is effectively a team lead, directing a team of AI agents with specialized capabilities across the entire software development lifecycle.

This keynote, drawing on practical experience serving millions of users through Tabnine, examines the fundamental challenges in making this vision a reality — and presents lessons learned from deploying AI-driven development tools at scale.

The Four Challenges

1. SDLC Transformation

How does the development lifecycle change when autonomous agents can handle entire tasks? What happens to code review, version control, and deployment when AI writes significant portions of the code?

2. Human-AI Interaction

What is the right way to communicate intent to an AI? How should developers consume AI-generated results? The interface between human and machine must be redesigned for each SDLC task.

3. Contextual Awareness

An LLM is an “ignorant genius” — capable but contextually unaware. Making AI hyper-local, tailored to your codebase, conventions, and domain, is the key to useful suggestions.

4. Trust & Verification

How can we trust AI-generated code, tests, and documentation? Establishing confidence requires transparency, validation mechanisms, and understanding the AI’s limitations.

Interactive Demo: AI Across the SDLC

Click on each stage of the software development lifecycle to see how AI transforms it — from traditional workflows to AI-augmented processes.

SDLC Pipeline Explorer

📝Design
💻Code
🔎Review
Test
📄Document
🚀Deploy

The Fundamental Theorem of Generative AI

When does it make sense to use generative AI for a task? The answer comes down to a simple cost equation. AI assistance pays off when the effort of specifying what you want plus verifying the result is significantly less than doing the work yourself. But here is the key insight: when the equation seems unfavorable, it is usually because the specification is partial — the developer has crucial context that never made it into the prompt.

When Does AI Pay Off?

cost(specify) + cost(verify) « cost(do it manually)

Select a task to see how the cost equation plays out. For routine tasks, specifications are naturally complete. For harder tasks, the prompt is only the tip of the iceberg — the real specification is the implicit context the AI doesn’t have.

Context Is Everything

An LLM without context about your project is like a brilliant new hire on their first day — technically capable but unfamiliar with the codebase, conventions, and domain. The key to practical AI assistance is onboarding the AI to your organization: feeding it your code patterns, style guides, internal libraries, and domain knowledge.

Toggle context sources below to see how additional context transforms an AI suggestion from generic boilerplate into code that fits your project.

Context-Aware Suggestions

Task: “Add error handling to this API endpoint.” Toggle context sources to see how the suggestion improves.

Without Context
With Context

Every Engineer Is a Team Lead

The most profound shift is organizational. When AI agents can handle code generation, test writing, documentation, and code review, the developer’s role evolves from writing all the code to directing a team of AI specialists. This requires new skills: clearly articulating intent, breaking down tasks for delegation, evaluating AI output, and knowing when to intervene.

The promise of AI-driven development is not to replace developers but to amplify them — turning every engineer into a team lead who directs AI agents across the SDLC, focusing human creativity on architecture, design decisions, and the problems that matter most.

Lessons from the Field

Deploying AI-driven development tools to millions of users at Tabnine reveals several practical lessons:

@inproceedings{yahav2023towards, title={Towards AI-Driven Software Development: Challenges and Lessons from the Field (Keynote)}, author={Yahav, Eran}, booktitle={Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)}, year={2023}, publisher={ACM} }