Technical Data Engineer

Moptra Infotech; Dadri, Ghaziabad District, Delhi; Hybrid; INR 2,500,000 - 4,500,000; Full time

Job summary

Moptra Infotech is seeking a hands-on Technical Data Engineer to design, build, and maintain production-grade data pipelines on Databricks. You will work with PySpark, Python, SQL and AWS to integrate data from databases, S3, files, and REST APIs, and implement Delta Lake, Unity Catalog and Medallion Architecture.

The role requires 5+ years in data engineering, 2+ years on Databricks, and a proactive approach to performance, reliability, and data quality in a hybrid work setup in Noida.

Qualifications

• Strong hands-on expertise in Python, PySpark, and Advanced SQL.
• Strong experience with the Databricks Lakehouse Platform.
• Hands-on knowledge of Unity Catalog, Delta Lake, Databricks Jobs & Workflows, Databricks Clusters, Notebooks, Repos, Medallion Architecture.
• Experience with REST API integration for data ingestion and data export.
• Strong understanding of ETL/ELT, batch processing, incremental loading, and data transformation.
• Experience with data modeling including Star Schema, Snowflake Schema, Fact & Dimension tables, and SCD concepts.
• Good understanding of data warehousing concepts and best practices.
• Experience handling structured and semi-structured data such as CSV, JSON, Parquet, and Delta.
• Knowledge of partitioning, file optimization, Spark performance tuning, and query optimization.
• Experience with Git and CI/CD practices.
• Good understanding of data quality, monitoring, troubleshooting, and production support.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL.
• Integrate data from multiple sources including databases, Amazon S3, files, and REST APIs.
• Build and manage data pipelines using Databricks Unity Catalog.
• Implement business logic, data transformations, and dimensional data models.
• Create, schedule, monitor, troubleshoot, and optimize Databricks Jobs and Workflows.
• Design and manage Delta Lake tables using Bronze, Silver, and Gold layers.
• Implement data quality checks, validations, error handling, logging, and monitoring.
• Optimize Spark workloads for performance, scalability, and reliability.
• Handle batch processing, incremental data loading, and large-volume data processing.
• Collaborate with engineering, business, and cross-functional teams to deliver production-ready data solutions.
• Follow best practices for version control, CI/CD, testing, documentation, and deployment.

Skills

Python

PySpark

SQL

Databricks

Unity Catalog

Delta Lake

Databricks Jobs & Workflows

Databricks Clusters

REST API integration

ETL/ELT

Data modeling

Star Schema

Snowflake Schema

SCD concepts

Git

CI/CD pipelines

Spark performance tuning

Partitioning

Parquet

JSON

CSV

Azure/AWS

Education

Bachelor's degree in Computer Science / Engineering / IT

Tools

Git

CI/CD pipelines

Job description

Technical Data Engineer Databricks

Location: Noida
Work Mode: Hybrid
Experience: 5+ Years
Relevant Databricks Experience: 2 to 5 Years
Employment Type: Full-Time

Job Summary

We are looking for a skilled Technical Data Engineer with strong hands-on experience in Databricks, Python, PySpark, SQL, and AWS to design, develop, and maintain scalable data engineering solutions. The ideal candidate will have experience building production-grade ETL/ELT pipelines, integrating data from databases, Amazon S3, files, and REST APIs, and working with Delta Lake, Unity Catalog, Medallion Architecture, and Databricks Workflows.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL.
• Integrate data from multiple sources including databases, Amazon S3, files, and REST APIs.
• Build and manage data pipelines using Databricks Unity Catalog.
• Implement business logic, data transformations, and dimensional data models.
• Create, schedule, monitor, troubleshoot, and optimize Databricks Jobs and Workflows.
• Design and manage Delta Lake tables using Bronze, Silver, and Gold layers.
• Implement data quality checks, validations, error handling, logging, and monitoring.
• Optimize Spark workloads for performance, scalability, and reliability.
• Handle batch processing, incremental data loading, and large-volume data processing.
• Collaborate with engineering, business, and cross-functional teams to deliver production-ready data solutions.
• Follow best practices for version control, CI/CD, testing, documentation, and deployment.

Required Technical Skills

• Strong hands-on expertise in Python, PySpark, and Advanced SQL.
• Strong experience with the Databricks Lakehouse Platform.
• Hands-on knowledge of:
• Unity Catalog
• Delta Lake
• Databricks Jobs & Workflows
• Databricks Clusters
• Notebooks
• Repos
• Medallion Architecture
• Experience with REST API integration for data ingestion and data export.
• Strong understanding of ETL/ELT, batch processing, incremental loading, and data transformation.
• Experience with data modeling, including Star Schema, Snowflake Schema, Fact & Dimension tables, and SCD concepts.
• Good understanding of data warehousing concepts and best practices.
• Experience handling structured and semi-structured data such as CSV, JSON, Parquet, and Delta.
• Knowledge of partitioning, file optimization, Spark performance tuning, and query optimization.
• Experience with Git and CI/CD practices.
• Good understanding of data quality, monitoring, troubleshooting, and production support.

Good to Have

• Experience with Databricks Auto Loader.
• Knowledge of Spark Declarative Pipelines.
• Experience with Kafka, Airflow, or dbt.
• Databricks certification.
• Experience working in Agile development environments.

Experience & Qualifications

• 5+ years of overall experience in Data Engineering.
• Minimum 2+ years of hands-on experience with Databricks.
• Strong programming and analytical skills.
• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field preferred.

What We are Looking For

The ideal candidate should be a hands-on Data Engineer who can independently build, optimize, troubleshoot, and maintain production-grade data pipelines on Databricks. Strong practical experience in PySpark + SQL + Databricks + AWS + REST APIs is essential.

Location: Noida
Work Mode: Hybrid