Senior AI Solution Engineer with MS Foundry, MCP

EPAM Systems; Turkey; On-site; TRY 300,000 - 600,000; Full time

Job summary

EPAM Systems Inc is seeking a Senior AI Solution Engineer in Turkey to design and build automated data ingestion pipelines on Azure Databricks, leveraging MS Foundry and MCP expertise. You will gather structured and unstructured data, automate workflows, and expose clean data models to enterprise AI layers.

Responsibilities include crafting high-throughput ingestion, implementing schema enforcement, and deploying pipelines via Azure DevOps and DABs to support AI agents across AI Search and GenAI

Qualifications

• 5+ years building high-volume data ingestion pipelines using Azure Databricks, PySpark and Auto Loader.
• Hands-on experience developing data scraping routines for unstructured data sources.
• Experience structuring ingested datasets using Open Knowledge Format within Databricks semantic layers.

Responsibilities

• Build high-throughput data ingestion pipelines and data scraping engines using Python, PySpark, and Databricks Auto Loader to collect unstructured document content, external registry feeds, and enterprise data.
• Structure and expose ingested data within the Databricks semantic layer, organize datasets using Open Knowledge Format standards for seamless consumption by AI Search, Databricks Genie Spaces, and LLM reasoning engines.
• Automate ingestion pipeline deployments, scraping job schedules, and code releases using Azure DevOps and Databricks Asset Bundles (DABs).
• Scrape and ingest healthcare and insurance data sources, including regulatory databases, public registries, and unstructured medical/claims files, to support AI agent workflows.
• Implement schema enforcement, automated ingestion failure alerts, and job telemetry within Databricks Workflows and Azure DevOps.

Skills

Azure Databricks

PySpark

Data ingestion pipelines

Data scraping

Open Knowledge Format

Databricks Asset Bundles (DABs)

Azure DevOps

English (B2)

Tools

Databricks Auto Loader

Databricks

Azure DevOps

Job description

We are seeking a Senior AI Solution Engineer with expertise in MS Foundry and MCP to build automated data ingestion pipelines and data scraping mechanisms on Azure Databricks that fuel the enterprise AI platform and semantic layer. This role focuses on gathering structured and unstructured data, automating workflows via Azure DevOps CI/CD pipelines, and exposing clean data models formatted via Open Knowledge Format directly to Databricks semantic layers, AI Search, and AI agent runtimes.ResponsibilitiesBuild high-throughput data ingestion pipelines and data scraping engines using Python, PySpark, and Databricks Auto Loader to collect unstructured document content, external registry feeds, and enterprise dataStructure and expose ingested data within the Databricks semantic layer, organize datasets using Open Knowledge Format standards for seamless consumption by AI Search, Databricks Genie Spaces, and LLM reasoning enginesAutomate ingestion pipeline deployments, scraping job schedules, and code releases using Azure DevOps and Databricks Asset Bundles (DABs)Scrape and ingest healthcare and insurance data sources, including regulatory databases, public registries, and unstructured medical/claims files, to support AI agent workflowsImplement schema enforcement, automated ingestion failure alerts, and job telemetry within Databricks Workflows and Azure DevOpsRequirementsExtensive experience (5+ years) building high-volume data ingestion pipelines using Azure Databricks, PySpark, and Auto LoaderHands-on experience developing data scraping routines and extraction engines for complex, unstructured data sourcesProven ability to structure ingested datasets using the Open Knowledge Format within Databricks semantic layers and AI Search for downstream GenAI consumptionTrack record of managing Databricks ingestion workflows and deployment pipelines using Azure DevOps and Databricks Asset Bundles (DABs)English proficiency at B2 level or higherWe offerCONTINUOUS UPSKILLING, LEARNING & DEVELOPMENTDiversity of tasks and projectsAssessment center for objective review of competency levelPersonal development planMentoring programs and leadership developmentCertification and professional development supportAccess to learning platforms including more than 2,500 internal coursesEnglish courses taught by certified teachersCORPORATE BENEFITSExtra leave daysReferral bonusesCOMPENSATION PACKAGECompetitive compensation paid in USDRegular salary and performance reviewsMEDICAL & HEALTHCAREPrivate health insuranceWell-being eventsWORKING ENVIRONMENTRecreation areas and kitchensTea, coffee and snacksSports equipment and game consolesIT EquipmentMicrosoft’s Software Assurance Home Use Program (HUP)Please note that our Talent Attraction Team reviews applications and CVs submitted in English.EPAM is global leader in AI transformation engineering and integrated consulting, serving Forbes Global 2000 companies and ambitious startups. With over thirty years of expertise in custom software, product and platform engineering, we empower our clients to become AI-Native enterprises, driving measurable value from innovation and digital investments.