Senior Data Engineer [T500-29047]

ADM; Bengaluru; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

ADM India Hub seeks a Senior Data Engineer to build and optimize data pipelines in Azure Databricks using Python and PySpark. You will implement Databricks Workflows, Asset Bundles, and CI/CD with GitHub to deliver scalable data solutions.

The role focuses on Data Warehousing/Data Mart design within Databricks, ensuring high-performance storage and efficient transformations across sources. Collaboration with data scientists and stakeholders is expected.

Qualifications

• Strong Python and Spark programming experience.
• Hands-on with Azure Databricks and PySpark.
• Experience building and optimizing data pipelines.
• Knowledge of data warehousing and data marts in Databricks.
• Familiarity with CI/CD workflows using GitHub.

Responsibilities

• Develop and maintain scalable data pipelines using Spark in Azure Databricks.
• Design and implement Databricks Workflows to automate pipelines.
• Create and manage Databricks Asset Bundles for reuse across teams.
• Build CI/CD pipelines for notebooks, jobs, and libraries with GitHub Actions.
• Design Data Warehousing/Data Mart solutions in Databricks and optimize performance.
• Collaborate with data scientists and analysts; mentor junior engineers.

Skills

Python

Apache Spark

PySpark

Data Pipelines

Data Modeling

Tools

Azure Databricks

GitHub Actions

Databricks Asset Bundles

Kafka

Hadoop

Flink

Job description

About ADM:

We are one of the world’s largest nutrition companies and a global leader in human and animal nutrition. We unlock the power of nature to provide nourishing quality of life by transforming crops into ingredients and solutions for foods, beverages, supplements, livestock, aquaculture, and pets.

About ADM India Hub:

At ADM, we have long recognized the strength and potential of India’s talent pool, which is why we have maintained a presence in the country for more than 25 years. Building on this foundation, we have now established ADM India Hub, our first GCC in India.

At ADM India Hub, we are hiring for IT and finance roles across diverse technology and business functions. We stand at the intersection of global expertise and local excellence, enabling us to drive innovation and support our larger purpose of unlocking the power of nature to enrich quality of life.

Job Title: Senior Data Engineer

Overview:

We are seeking a skilled and motivated Azure Databricks Data Engineer to join our dynamic team. The ideal candidate will have strong experience with Python, Spark programming, and expertise in building and optimizing data pipelines in Azure Databricks. You will play a pivotal role in leveraging Databricks workflows, Databricks Asset Bundles, and CI/CD pipelines using GitHub to deliver high-performance data solutions. A solid understanding of Data Warehousing and Data Mart architecture in Databricks is critical for success in this role. If you’re passionate about data engineering, cloud technologies, and scalable data architecture, we’d love to hear from you!

Key Responsibilities:

Python and Spark Programming:

• Develop and maintain scalable data pipelines using Apache Spark within Azure Databricks/ Python/ PySpark.
• Write optimized, high-performance Spark jobs to process large volumes of data efficiently.
• Utilize PySpark for distributed data processing, transformation, and aggregation tasks.

Databricks Workflows:

• Design and implement Databricks Workflows to automate data pipeline execution, orchestrating complex workflows and batch jobs.
• Set up task dependencies, triggers, and notifications to ensure smooth and reliable execution.
• Monitor, troubleshoot, and optimize Databricks workflows for optimal performance and minimal failures.

Databricks Asset Bundles:

• Create and manage reusable components such as Databricks Asset Bundles, including notebooks, libraries, and models.
• Share and reuse asset bundles across teams to increase efficiency and ensure consistency in development.

CI/CD for Databricks Artifacts using GitHub:

• Implement CI/CD pipelines using GitHub Actions for the continuous integration and deployment of Databricks notebooks, jobs, and libraries.
• Automate the testing, building, and deployment processes to ensure smooth, consistent code delivery across environments.
• Collaborate with teams to implement version control practices and code reviews using GitHub.

Data Warehousing & Data Mart Design:

• Design and implement Data Warehousing and Data Mart solutions using Databricks, ensuring high-performance storage and retrieval of structured data.
• Integrate data from multiple sources into a central data warehouse using Spark-based transformations, ensuring efficient schema design and query performance.
• Implement dimensional modeling, including star and snowflake schemas, within Azure Databricks for data marts to support business intelligence and reporting.
• Data Pipeline Optimization and Management:
• Continuously monitor and optimize Databricks-based data pipelines for performance, scalability, and cost efficiency.
• Implement best practices for data partitioning, caching, and query optimization within the Databricks platform.
• Troubleshoot and resolve issues related to data integrity, performance, and workflow execution.

Collaboration and Stakeholder Communication:

• Work closely with data scientists, analysts, and other teams to understand requirements and build data solutions that meet business needs.
• Communicate technical concepts effectively to both technical and non-technical stakeholders.
• Provide mentorship and guidance to junior data engineers on Databricks best practices, data architecture, and efficient coding techniques.

Preferred Qualifications:

Cloud Certifications:

• Azure certifications, particularly in Databricks, Data Engineering, or Cloud Solutions, are a plus.

Big Data Technologies:

• Familiarity with other big data tools such as Kafka, Hadoop, or Flink for streaming and real-time data processing is a plus.

Data Science/ML Experience:

• Exposure to machine learning workflows and model management within Databricks (e.g., using MLflow) is beneficial.