Azure Data Engineer

APN Consulting, Inc.; Mumbai; On-site; INR 1,500,000 - 2,300,000; Full time

Job summary

APN Consulting, Inc. in Mumbai is seeking a Data Engineer for Rialto with 6-8 years of experience. You will work on AI-driven data platforms, ML/DL models, and enterprise-scale data solutions.

The role emphasizes Databricks, PySpark, NLP, and cloud integrations to build scalable data lakehouse architectures and robust data pipelines across multiple business domains.

Qualifications

• Proficiency in Python, TensorFlow, PyTorch, and other popular AI libraries.
• Experience with Databricks and PySpark for data engineering.
• Experience with cloud platforms (AWS, Azure, Google Cloud) and data lakehouse concepts.

Responsibilities

• Machine Learning and Deep Learning Expertise — Strong foundation in ML and DL concepts, including transformers, RNNs, and LSTMs, with training methods.
• Strategic Data Solution Development — Contribute to building a data platform serving as the single source of truth data lakehouse.
• Design and Development — Architect data solutions, integrate Databricks, design ELT processes and data models.
• Data Management and Optimization — Optimize storage/processing in Databricks and implement scalable lakehouse solutions.
• Data Engineering Expertise — Lead development of data tables across domains, ensure third-party data ingestion is well-structured.
• Strategic Integration and Security — Collaborate with security/infrastructure for secure, end-to-end analytics.

Skills

Python

PySpark

NLP

LLMs

Spark

Tools

Databricks

Azure Data Factory

TensorFlow

PyTorch

Job description

Role Name

Data Engineer for Rialto

Experience

6 to 8 Years

Primary Location

Mumbai

Other Acceptable Locations

Kolkata, Pune, Hyderabad, Bangalore, Chennai, Noida

Key Skills

Python, PySpark, Microsoft Fabric, AI & Gen AI - Products & Tools, Azure Data Factory

Must-Have Technical/Functional Skills

• Proficiency in Python, TensorFlow, PyTorch, and other popular libraries for AI development.

Experience Required

• Conversational AI and LLM Experience — Hands-on experience with conversational AI systems, including Retrieval-Augmented Generation (RAG) models, fine-tuning large language models (LLMs) at an enterprise scale, and developing dialogue management systems to handle dynamic user interactions effectively.
• NLP and Language Model Specialization — Proficiency with NLP techniques and a deep understanding of language models like GPT, BERT, T5, and other transformer-based models.
• Data Engineering and Analytics — Substantial experience in data engineering and analytics, with strong expertise in Databricks and PySpark. Hands-on knowledge of Apache Spark fundamentals, including Spark Architecture, its APIs, and how to leverage it for data processing and analysis.
• Programming and Query Skills — Proficiency in PySpark, Python, and SQL is essential for data manipulation, analysis, and transformation.
• Cloud Platforms — Demonstrated experience with cloud platforms (AWS, Azure, Google Cloud) and their integration with Databricks.
• Data Solutions Architecture — Experience as a data solutions architect is a significant advantage, demonstrating the ability to design comprehensive data solutions.
• Soft Skills — Excellent communication and team collaboration skills.

Roles & Responsibilities

• Machine Learning and Deep Learning Expertise — Strong foundation in ML and DL concepts, including model architectures such as transformers, RNNs, and LSTMs, along with training techniques like supervised, unsupervised, and reinforcement learning.
• Strategic Data Solution Development — Serve as a primary contributor to the development of data solutions with a strategic outlook, focused on building a data platform that serves as the single source of truth data lakehouse.
• Design and Development — Architect and manage data solutions, ensuring integration of Databricks with existing data systems. Design ELT processes and data models in line with business requirements.
• Data Management and Optimization — Optimize data storage and processing in Databricks. Implement data lakehouse solutions, ensuring performance and scalability.
• Data Engineering Expertise — Lead the development of data tables tailored to specific use cases by engineering critical elements from multiple data domains. Ensure ingestion of third-party data is well-structured, compliant with data quality standards, and traceable from the consumption layer back to the raw data layer.
• Strategic Integration and Security — Partner with the Information Security and Infrastructure teams to streamline data integration, maintain data security, and follow access best practices. Contribute to the creation of end-to-end data analytics solutions.