AI Product Analyst - AI Receptionist Team

At MEDvidi, we are scaling AI-powered systems across critical parts of our product and operations.

As these systems become more autonomous and deeply integrated into user workflows, understanding how they behave in production becomes a core business challenge.

We are hiring a Product Analyst to help us evaluate, monitor, and improve AI agents and LLM-powered workflows operating at scale.

This role is focused on analyzing real AI system behavior in production environments:

• understanding product and operational impact,

• identifying failure patterns on generational tasks,

• measuring quality and reliability of outputs,

• helping teams make accurate decisions under ambiguity.

Responsibilities:

• Analyze production behavior of AI agents, automation systems, and LLM-powered workflows,

• Investigate hallucinations, inconsistencies, failure modes, and quality degradation across automated user journeys,

• Work directly with logs, conversations, raw system outputs, and operational datasets to identify hidden patterns and actionable insights,

• Design evaluation approaches and analytical frameworks for measuring AI system quality and reliability,

• Build monitoring systems and dashboards for AI performance visibility,

• Identify operational and product risks introduced by automation,

• Measure the impact of AI systems on business metrics, operational efficiency, and user experience,

• Detect data quality issues, shifts, and anomalies affecting AI system performance,

• Partner closely with Product, Engineering, QA, and Operations teams on AI-related initiatives,

• Support rapid decision-making in fast-changing and ambiguous environments.

Requirements:

• 3+ years of experience in analytical roles related to AI systems, ML products, automation platforms, or complex production systems,

• Strong Python skills for deep analytical investigations and working with large-scale noisy text data,

• Advanced SQL skills,

• Understanding of LLM limitations,hallucinations, evaluation challenges, prompt sensitivity, model trade-offs, and AI failure modes,

• Experience building monitoring systems, analytical frameworks, or operational dashboards for production systems,

• Ability to work with incomplete, ambiguous, and rapidly changing data environments,

• Strong ownership mindset and ability to independently drive investigations and analytical conclusions,

• Ability to connect AI system quality with business and operational impact,

• Fluent Russian; English level B2 or higher,

• Ability to work CET-overlapping hours.

Nice to Have:

• Experience with conversational AI or AI agents,

• Experience evaluating LLM outputs or annotation quality,

• Experience with AI quality metrics and evaluation methodologies,

• Experience working with Langchain infrastructure,

• Experience with ML data pipelines or synthetic data generation,

• Background in anomaly detection.

What we offer:

• Work on a modern AI product where quality is a strategic function, not a formality

• Real influence on product direction and engineering practices

• High-impact work in a product-driven medical tech company

• Startup environment: fast iteration, high ownership, minimal bureaucracy

• Competitive compensation and growth opportunities

Benefits:

• Health insurance after probation period

• Sports & wellness compensation

• Fully remote, long-term B2B collaboration

• 19 working days of vacation per year

• 3 additional wellness days per year

• Paid sick leave for the first 5 working days

• Gifts to employees for important events

• Career Development

• Personalized English lessons via Preply

Ready to make an impact? Send us your profile - let’s build something meaningful together.