Master Thesis: Observability and Analytics for AI-Assisted Development

Ericsson; Lunds kommun; Vor Ort; SEK 223.000 - 290.000; Vollzeit

Zusammenfassung

Ericsson in Lund invites a student to work on an analytics and observability framework for an AI-assisted command-line development tool. The project will instrument the tool, collect telemetry, and provide actionable insights into model choices, token use, latency, and costs.

You will design measurements, analyze usage patterns, and validate recommendations through controlled studies, contributing to an open, privacy-respecting data platform that helps developers understand and optimize their AI

Qualifikationen

• Knowledge in Computer Science, Software Engineering, Data Science, or related field.
• Experience with at least one programming language, preferably Python, JavaScript, or TypeScript.
• Interest in observability, analytics, distributed systems, or AI developer tools.
• Familiarity with data collection, visualization, experimentation, or statistical analysis is beneficial.
• Ability to work independently and communicate findings clearly.

Aufgaben

• Identify signals and metrics for understanding AI-assisted development workflows.
• Design and implement instrumentation to capture relevant events with privacy in mind.
• Analyze data to identify cost and latency drivers and propose optimizations.
• Validate recommendations through controlled cohorts or user studies.
• Communicate findings and collaborate with engineers and researchers.

Kenntnisse

Observability concepts

Python

JavaScript/TypeScript

Data analysis

Independent work

Ausbildung

Computer Science/Software Engineering/Data Science

Tools

Telemetry tooling

Dashboard development

Jobbeschreibung

About This Opportunity

AI-assisted command-line development tools are changing how software is built. They can select different models, consume varying amounts of tokens and credits, compact conversation context, retry failed requests, and delegate work to sub-agents. However, developers often have limited visibility into what is happening behind the scenes—and how these behaviours affect cost, latency, reliability, and the overall user experience.

Join our Team

This thesis will explore how to build an analytics and observability framework for an AI-assisted command-line development tool. The framework will make the tool’s internal behaviour measurable and understandable, helping developers see how tokens and credits are consumed and how different workflows influence performance and cost. The work will also investigate whether data-driven recommendations can improve the tool’s efficiency without compromising the developer experience.

What you will do

The thesis will begin by identifying the most valuable signals and metrics for understanding AI-assisted development workflows. Our flag AI agent is Kiro-CLI. You will design and implement an instrumentation and analytics framework that captures relevant events while respecting privacy and usability requirements.

The framework may include measurements such as:

• Model selection and model switching
• Token and credit consumption
• Context size and context-compaction events
• Retries, failures, and recovery behaviour
• Sub-agent creation, orchestration, and execution time
• End-to-end and component-level latency
• User interactions and perceived experience

You will then use the collected data to analyse common usage patterns, identify sources of unnecessary cost or delay, and develop actionable optimization recommendations. Potential recommendations could include selecting a more suitable model, improving context-management strategies, reducing redundant retries, or changing sub-agent orchestration policies.

Finally, you will validate selected recommendations through a controlled cohort or user study. The evaluation will investigate whether the recommendations reduce token or credit consumption, improve latency and reliability, and help developers better understand and control their AI usage.

The exact scope can be adapted based on your interests and the maturity of the tool. Possible areas of exploration include:

• Designing a useful observability model for agentic command-line tools.
• Creating dashboards or in-tool feedback for token and credit consumption.
• Comparing different strategies for context compaction.
• Evaluating model-routing and model-selection policies.
• Measuring the impact of retries and sub-agent orchestration on cost and latency.
• Defining user-experience metrics for AI-assisted software development.
• Investigating privacy-preserving collection and aggregation of developer telemetry.

The skills you bring

Knowledge in Computer Science, Software Engineering, Data Science, or a related field

Experience with at least one programming language, preferably Python, JavaScript, TypeScript, or a similar language

Interest in observability, analytics, distributed systems, developer tools, or artificial intelligence.

Familiarity with data collection, visualization, experimentation, or statistical analysis is beneficial.

Ability to work independently, investigate open-ended problems, and communicate findings clearly.

Bonus skills

Experience with command-line tools or developer productivity platforms.

Experience with large language models, AI agents, or multi-agent systems.

Knowledge of telemetry, event-driven systems, metrics, tracing, or dashboard development.

Experience designing user studies or controlled experiments.

Why join Ericsson?

At Ericsson, you´ll have an outstanding opportunity. The chance to use your skills and imagination to push the boundaries of what´s possible. To build solutions never seen before to some of the world’s toughest problems. You´ll be challenged, but you won’t be alone. You´ll be joining a team of diverse innovators, all driven to go beyond the status quo to craft what comes next.

Encouraging a diverse and inclusive organization is core to our values at Ericsson, that's why we champion it in everything we do. We truly believe that by collaborating with people with different experiences we drive innovation, which is essential for our future growth. We encourage people from all backgrounds to apply and realize their full potential as part of our Ericsson team. Ericsson is proud to be an Equal Opportunity Employer. learn more.

Primary country and city:

Sweden (SE) || Lund