Business Data Insights Senior Specialist

maersk; Chennai District; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Maersk is seeking a Data & AI Engineer to own the end-to-end data-to-AI lifecycle, building robust pipelines with Azure Data Lake, Data Factory and Databricks, and deploying ML and generative AI models on Azure ML and Azure OpenAI.

You will translate business requirements into scalable data and AI solutions, collaborate with stakeholders, data scientists, BI and product teams, and deliver measurable value from raw data to production-ready AI features at scale.

Qualifications

• 5+ years of combined data engineering and AI/ML engineering experience.
• Experience building scalable data pipelines in cloud environments.
• Experience with ETL/ELT development and data lake architectures.

Responsibilities

• Design, develop, test, deploy, and maintain scalable data pipelines using Azure Data Factory, Azure Data Lake, Azure Databricks, SQL, and related technologies.
• Build automated ETL/ELT pipelines to ingest, transform, validate, and publish data for analytics and AI model consumption.
• Design, build, and fine-tune machine learning and generative AI models to solve business problems using the curated data pipelines as the foundation.
• Develop RAG pipelines and integrate LLM-based solutions with enterprise data sources and applications.
• Work with structured, semi-structured, and unstructured datasets, preparing them for analytics and model training/evaluation.
• Develop data and model transformation logic using SQL, PySpark, Spark SQL, Databricks notebooks, and Python ML frameworks.
• Create and maintain reliable data flows and AI services across raw, curated, and consumption-ready layers, including production model inference.

Skills

Data integration

ETL/ELT

Azure Data Factory

Azure Data Lake

Azure Databricks

SQL

Python

Azure ML

Azure OpenAI

RAG pipelines

Vector databases

PyTorch/TensorFlow

MLOps

Docker/Kubernetes

Education

Tools

Power BI

Git

CI/CD

VA Cloud infra

Job description

Job Description

We are looking for a skilled and versatile Data & AI Engineer who can own the full data-to-AI lifecycle - designing and building robust data pipelines using Azure Data Lake, Azure Data Factory, and Azure Databricks, and then using that data to build, fine-tune, and deploy machine learning and generative AI solutions with Azure Machine Learning and Azure OpenAI.

The role requires strong hands-on experience across both disciplines: developing ETL/ELT pipelines, transforming structured and semi-structured data, and ensuring reliable data availability, as well as building RAG pipelines, integrating LLM-based solutions, and operationalizing models with MLOps best practices.

The candidate should be comfortable working with business stakeholders, data scientists, BI teams, product teams, and technical teams to understand requirements, design scalable data and AI solutions, and deliver measurable business value end to end - from raw data to production-ready AI features.

Role: Data & AI Engineer

Work Experience

5+ years of combined experience in data engineering and AI/ML engineering, including building scalable data pipelines and cloud data platforms as well as developing and deploying machine learning or generative AI solutions, in an Agile or DevOps environment.

Role Summary

We are looking for a skilled and versatile Data & AI Engineer who can own the full data-to-AI lifecycle - designing and building robust data pipelines using Azure Data Lake, Azure Data Factory, and Azure Databricks, and then using that data to build, fine-tune, and deploy machine learning and generative AI solutions with Azure Machine Learning and Azure OpenAI.

The role requires strong hands-on experience across both disciplines: developing ETL/ELT pipelines, transforming structured and semi-structured data, and ensuring reliable data availability, as well as building RAG pipelines, integrating LLM-based solutions, and operationalizing models with MLOps best practices.

The candidate should be comfortable working with business stakeholders, data scientists, BI teams, product teams, and technical teams to understand requirements, design scalable data and AI solutions, and deliver measurable business value end to end - from raw data to production-ready AI features.

Key Skills

• 4+ years of experience spanning data engineering and AI/ML engineering, with strong exposure to data integration, ETL/ELT development, and cloud-based data and AI platforms.
• Hands-on experience with Azure Data Factory for building, scheduling, monitoring, and managing data pipelines.
• Working knowledge of Azure Data Lake Storage for storing, organizing, and managing large volumes of data.
• Experience with Azure Databricks, including notebooks, Spark SQL, PySpark, data transformation, and performance optimization.
• Strong SQL and Python skills for querying, transformation, data validation, troubleshooting, and performance tuning.
• Hands-on experience with Azure Machine Learning or Azure AI Studio for training, deploying, and managing ML models.
• Practical experience with Azure OpenAI Service or similar LLM platforms, including prompt engineering, fine-tuning, and model integration.
• Experience building RAG pipelines using vector databases (e.g., Azure AI Search, FAISS, Pinecone) for grounding LLM responses in enterprise data.
• Experience with ML/AI frameworks such as PyTorch, TensorFlow, scikit-learn, LangChain, or Semantic Kernel.
• Good understanding of data lake architecture (bronze, silver, gold / medallion architecture) and MLOps practices (model versioning, CI/CD for ML, monitoring, retraining).
• Experience with data modelling, data quality checks, data profiling, reconciliation, feature engineering, and model evaluation.
• Exposure to DevOps practices such as Git, CI/CD, containerization (Docker, Kubernetes), version control, and deployment processes.
• Working knowledge of Power BI or similar reporting tools, and knowledge of legacy ETL tools (Informatica, SSIS, Teradata, Oracle) will be an added advantage.

Key Responsibilities

• Design, develop, test, deploy, and maintain scalable data pipelines using Azure Data Factory, Azure Data Lake, Azure Databricks, SQL, and related technologies.
• Build automated ETL/ELT pipelines to ingest, transform, validate, and publish data for analytics, reporting, and AI model consumption.
• Design, build, and fine-tune machine learning and generative AI models to solve business problems, using the curated data pipelines as the foundation.
• Develop RAG pipelines and integrate LLM-based solutions with enterprise data sources and applications.
• Work with structured, semi-structured, and unstructured datasets, preparing them for both analytics and model training/evaluation.
• Develop data and model transformation logic using SQL, PySpark, Spark SQL, Databricks notebooks, and Python ML frameworks.
• Create and maintain reliable data flows and AI services across raw, curated, and consumption-ready layers, including production model inference.

Per