Data Engineer - Databricks

Alliance Recruitment Agency; Chennai District; On-site; INR 2,500,000 - 3,500,000; Full time

Job summary

Alliance Recruitment Agency is seeking an experienced data engineer to design and implement Databricks-based data solutions in Chennai. You will work with data engineers, analysts and scientists to build scalable data architectures, pipelines and models that meet business goals.

The role requires strong SQL, Python, Airflow and data warehousing expertise, with a focus on security, compliance and performance optimization in a Databricks environment.

Qualifications

• 5+ years of hands-on data engineering and analytics experience.
• Proficiency in Databricks with Spark-based data processing.
• Strong knowledge of data modeling, ETL, and data pipeline development.
• Advanced SQL for data querying, transformation, and analysis.
• Expertise in Python for automation and data manipulation in Databricks.
• Extensive experience with Apache Airflow for ETL orchestration.
• Familiarity with big data tech such as Spark, Hadoop, and related frameworks.
• Strong data security and compliance practices.
• Proficient in data warehousing concepts and modeling.
• Experience using Git for version control.

Responsibilities

• Collaborate with stakeholders to gather data requirements and design Databricks-based solutions aligned with business goals.
• Engineer and document scalable, secure, high-performance data architectures and models.
• Guide cross-functional teams including data engineers, analysts, and scientists on best practices.
• Develop and manage complex data pipelines, ETL workflows and integration strategies within Databricks.
• Optimize and monitor Databricks clusters, workloads and queries for performance and cost efficiency.
• Implement strict security controls and ensure compliance with data privacy standards within Databricks solutions.
• Create and maintain comprehensive technical documentation including architecture diagrams and data flow representations.
• Act as the primary technical contact, providing updates to stakeholders and ensuring project progress communications.
• Collaborate with cross-functional teams to ensure end-to-end data solutions meet project objectives.
• Stay updated with Databricks trends and implement new methodologies and tools.

Skills

Databricks

Spark

SQL

Python

Apache Airflow

Data Modeling

ETL

Data Warehousing

Security & Compliance

Git

Tools

Databricks

Apache Airflow

Hadoop

Git

Job description

Key Responsibilities:

• Collaborate with stakeholders to gather data requirements and design Databricks-based solutions aligned with business goals.
• Engineer and document scalable, secure, and high-performance data architectures and models.
• Provide expert guidance to cross‑functional teams including data engineers, analysts, and scientists, ensuring adherence to best practices.
• Develop and manage complex data pipelines, ETL workflows, and data integration strategies within Databricks.
• Optimize and monitor Databricks clusters, workloads, and queries for performance and cost efficiency.
• Implement strict security controls and ensure compliance with data privacy standards within Databricks solutions.
• Create and maintain comprehensive technical documentation including architecture diagrams, data flow representations, and guides.
• Act as the primary technical point of contact, providing updates to stakeholders and ensuring smooth communication of project progress.
• Collaborate closely with cross‑functional teams to ensure end‑to‑end data solutions are met with project objectives.
• Stay up‑to‑date with trends in data engineering and Databricks, recommending and implementing new methodologies and tools.

Required Skills & Experience:

• Minimum of 5+ years of hands‑on experience in data engineering and analytics.
• Proficiency in Databricks, with a focus on Spark‑based data processing and analytics.
• Strong understanding of data engineering concepts, including data modeling, ETL processes, and data pipeline development.
• Advanced skills in SQL for querying, transforming, and analyzing data.
• Expertise in Python programming for automation and data manipulation within Databricks.
• Strong experience with Apache Airflow for orchestrating and scheduling ETL workflows.
• Familiarity with big data technologies such as Spark, Hadoop, and related frameworks.
• Strong understanding of data security and compliance best practices.
• Proficient in data modeling principles and data warehousing concepts.
• Experience using version control systems like Git.
• Excellent communication and collaboration skills, with the ability to work with cross‑functional teams.