Lead Data Engineer

CoffeeBeans

Bengaluru

On-site

INR 2,600,000 - 3,800,000

Full time

23 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

CoffeeBeans Consulting in Bengaluru is seeking a Data Engineer L2/L3 to design and optimize enterprise data pipelines, build scalable lakehouse architectures, and enable AI-driven analytics.

You will collaborate with clients, mentor engineers, and drive technical decisions across Databricks, Spark, Kafka, and cloud services, with 6–10 years of hands-on experience.

Qualifications

  • 6–10 years of data engineering experience.
  • Hands-on with Databricks Lakehouse and Delta Lake.
  • Strong Python and SQL skills; distributed data processing expertise.
  • Experience with Spark, Kafka, and streaming architectures.
  • Proficient in data modeling, ETL/ELT, and data governance.
  • Knowledge of cloud-native data services, especially AWS.

Responsibilities

  • Design and implement enterprise-grade Databricks Lakehouse architectures using Delta Lake and Unity Catalog.
  • Build scalable batch and real-time data ingestion pipelines using Lakeflow Connect, SDP, Auto Loader, Spark, and Kafka.
  • Implement CDC architectures using Debezium, Kafka/Kafka Connect, and relational databases (PostgreSQL, MySQL, SQL Server, Oracle).
  • Develop streaming and event-driven pipelines with Kafka, Spark Structured Streaming, and related tech.
  • Establish schema evolution and data contracts using Karapace / Schema Registry.
  • Implement centralized governance with Unity Catalog across catalogs, schemas, RBAC, and data access policies.
  • Develop metadata-driven ingestion, data quality, reconciliation, profiling, and observability solutions.
  • Design Bronze/Silver/Gold data layers and data modeling strategies for analytics.

Skills

Databricks Lakehouse
Python
SQL
Apache Spark
Kafka
Unity Catalog
Delta Lake
CI/CD
AWS
Data governance

Tools

Lakeflow Connect
SDP
Auto Loader
Debezium
Kafka Connect
Schema Registry
Databricks Asset Bundles
Jenkins
GitHub Actions

Job description

Experience: 6–10 years in data engineering. | Location: Bangalore. | Work Mode: On-site.

About Us

CoffeeBeans Consulting is a tech consulting firm focused on making organizations AI-ready by structuring their data efficiently across various sources and enabling AI-driven solutions. We specialize in data architecture, pipelines, governance, MLOps, and Gen AI solutions.

Role Overview

Join CoffeeBeans Consulting as a Data Engineer L2/L3 and immerse yourself in a transformative role where your expertise will directly contribute to the future of AI. Located in Bangalore, this position offers a unique opportunity to work at the forefront of data engineering, shaping the way businesses leverage their data to drive innovation. With 6-10 years of experience, you will play a pivotal role in building and optimizing scalable data pipelines that empower analytics and AI/ML solutions. This is not just a job; it’s a chance to elevate your career in a company that values engineering excellence and client impact.

Key Responsibilities
  • Design and implement enterprise-grade Databricks Lakehouse architectures using Delta Lake and Unity Catalog.
  • Build scalable batch and real-time data ingestion pipelines using Lakeflow Connect, SDP, Auto Loader, Spark, and Kafka.
  • Design and implement CDC architectures using Debezium, Kafka/Kafka Connect, and relational databasessuch as PostgreSQL, MySQL, SQL Server, and Oracle.
  • Implement streaming and event-driven data pipelines using Kafka, Spark Structured Streaming, and related technologies.
  • Design and manage schema evolution and data contracts using Karapace / Schema Registry.
  • Implement centralized governance using Unity Catalog, including catalogs, schemas, RBAC, row/column-level security, lineage, and data access policies.
  • Develop metadata-driven ingestion frameworks, data quality, reconciliation, profiling, and observability solutions.
  • Design Bronze, Silver, and Gold data layers and appropriate data modeling strategies for analytical workloads.
  • Establish engineering best practices covering CI/CD, testing, deployment, monitoring, logging, and operational support.
  • Use Databricks Asset Bundles (DAB) and CI/CD tools such as Jenkins/GitHub Actions for automated deployment.
  • Work with cloud services such as AWS S3, IAM, networking, monitoring, and security services.
  • Lead technical discussions with clients, translate business requirements into technical solutions, and drive architecture decisions.
  • Troubleshoot complex data engineering, CDC, streaming, performance, and production issues.
  • Mentor engineers and provide technical direction across data engineering initiatives.
Must-Have Skills
  • Strong hands-on experience with Databricks and Lakehouse architecture.
  • Experience with Snowflake and dbt.
  • Advanced Python and SQL skills.
  • Strong expertise in Apache Spark / PySpark and distributed data processing.
  • Hands-on experience with Unity Catalog and Delta Lake.
  • Experience with Lakeflow Connect, SDP / Spark Declarative Pipelines, and Auto Loader.
  • Strong understanding of CDC architectures using Debezium and Kafka.
  • Hands-on experience with Kafka / Kafka Connect.
  • Experience with Karapace or Schema Registry and schema evolution.
  • Strong understanding of ETL/ELT, data modeling, data warehousing, streaming, and data integration patterns.
  • Experience with production-grade data pipelines and orchestration.
  • Strong understanding of cloud-native data services, particularly AWS.
  • Experience with CI/CD and Databricks Asset Bundles (DAB).
  • Experience leading technical implementations and working directly with business/client stakeholders.
Good to Have
  • Experience with Apache Flink or other real-time processing frameworks.
  • Experience implementing data governance, lineage, security, data quality, and observability.
  • Experience with AWS S3, IAM, Glue, MSK/Kafka, and cloud networking.
  • Experience with AI/ML data platforms and GenAI workloads.
  • Databricks certifications, particularly Databricks Certified Data Engineer Professional.
Other Expectations
  • AWS Data Engineering/Data Analytics certifications.
  • Strong ownership and problem-solving mindset.
  • Ability to balance hands-on engineering with architecture and technical leadership.
  • Strong client-facing and communication skills.
  • Ability to mentor and guide engineering teams.
  • Willingness to adapt to new technologies and client environments.
  • Willingness to travel within India and internationally for short/medium-term client assignments.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Coffeebeans.io - Senior Data Engineer - Python
Coffeebeans.io - Senior Data Engineer - Python

CoffeeBeans • Bengaluru Urban

Hybrid
INR 1,500,000 - 2,800,000
Senior Data Engineer
Senior Data Engineer

Coffeebeans Consulting • Bengaluru

On-site
INR 3,000,000 - 6,000,000
Senior Data Engineer
Senior Data Engineer

CoffeeBeans • Bengaluru

On-site
INR 1,800,000 - 2,400,000
Resident Solution Architect – Databricks | 10+ Yrs | Bangalore
Resident Solution Architect – Databricks | 10+ Yrs | Bangalore

nHRMS Inc. • Bengaluru

Hybrid
INR 4,200,000 - 7,800,000
Resident Solution Architect – Databricks | 10+ Yrs | Bangalore
Resident Solution Architect – Databricks | 10+ Yrs | Bangalore

N Human Resources & Management Systems • Bengaluru

On-site
INR 2,500,000 - 3,500,000
Sr. Databricks Engineer
Sr. Databricks Engineer

ExlService Holdings, Inc. • Pune District

Hybrid
INR 3,500,000 - 6,500,000
Data Architect
Data Architect

Infosys • Bengaluru Urban

On-site
INR 1,200,000 - 2,000,000
Cutting-edge cloud and data technologies
Collaborative culture focusing on innovation
Cross-functional project visibility
Senior Databricks Engineer
Senior Databricks Engineer

Value Vision • Bengaluru

On-site
INR 4,200,000 - 6,600,000
Data Architect
Data Architect

United HR Solutions Pvt. Ltd. • India

On-site
INR 550,000 - 900,000
Databricks - Data Engineer
Databricks - Data Engineer

Tredence Inc. • Bengaluru

On-site
INR 1,500,000 - 2,500,000