Senior Data Engineer

CoffeeBeans; Bengaluru; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

CoffeeBeans Consulting is seeking a Data Engineer L2 in Bangalore to design scalable data pipelines and Lakehouse architectures, leveraging Delta Lake and Unity Catalog. You will build batch and streaming ingestion using Kafka, Spark, and related tools, and drive architecture decisions with clients.

The role emphasizes hands-on engineering, leadership, and mentoring, with the opportunity to work on end-to-end data platforms for analytics and AI/ML initiatives.

Qualifications

• Strong hands-on experience with Databricks and Lakehouse architecture.
• Advanced Python and SQL skills.
• Strong expertise in Apache Spark / PySpark and distributed data processing.
• Hands-on experience with Unity Catalog and Delta Lake.

Responsibilities

• Design and implement Databricks Lakehouse architectures using Delta Lake and Unity Catalog.
• Build scalable batch and real-time data ingestion pipelines using Lakeflow Connect, SDP, Auto Loader, Spark, and Kafka.
• Design and implement CDC architectures using Debezium, Kafka/Kafka Connect, and relational databases such as PostgreSQL, MySQL, SQL Server, and Oracle.
• Implement streaming and event-driven data pipelines using Kafka, Spark Structured Streaming, and related technologies.
• Design and manage schema evolution and data contracts using Karapace / Schema Registry.
• Implement centralized governance using Unity Catalog, including catalogs, schemas, RBAC, row/column-level security, lineage, and data access policies.
• Develop metadata-driven ingestion frameworks, data quality, reconciliation, profiling, and observability solutions.
• Design Bronze, Silver, and Gold data layers and appropriate data modeling strategies for analytical workloads.
• Establish engineering best practices covering CI/CD, testing, deployment, monitoring, logging, and operational support.
• Use Databricks Asset Bundles (DAB) and CI/CD tools such as Jenkins/GitHub Actions for automated deployment.
• Work with cloud services such as AWS S3, IAM, networking, monitoring, and security services.
• Lead technical discussions with clients, translate business requirements into technical solutions, and drive architecture decisions.
• Troubleshoot complex data engineering, CDC, streaming, performance, and production issues.
• Mentor engineers and provide technical direction across data engineering initiatives.

Skills

Databricks Lakehouse

Python

SQL

Apache Spark / PySpark

Unity Catalog

Delta Lake

Kafka

Kafka Connect

Debezium

Data modeling

ETL/ELT

Data warehousing

CI/CD

Databricks Asset Bundles

AWS

Lakehouse architecture

Mentoring

Client leadership

Education

Databricks Certified Data Engineer Professional

AWS Data Analytics Certification

Tools

Lakeflow Connect

SDP

Auto Loader

Spark Declarative Pipelines

Jenkins / GitHub Actions

Databricks

Unity Catalog

Delta Lake

Databricks Asset Bundles

Job description

Experience: 3-10 years in data engineering.

Role Overview

Join CoffeeBeans Consulting as a Data Engineer L2 and immerse yourself in a transformative role where your expertise will directly contribute to the future of AI. Located in Bangalore, this position offers a unique opportunity to work at the forefront of data engineering, shaping the way businesses leverage their data to drive innovation. With 4-7 years of experience, you will play a pivotal role in building and optimizing scalable data pipelines that empower analytics and AI/ML solutions. This is not just a job; it's a chance to elevate your career in a company that values engineering excellence and client impact.

Key Responsibilities

• Design and implement enterprise-grade Databricks Lakehouse architectures using Delta Lake and Unity Catalog.
• Build scalable batch and real-time data ingestion pipelines using Lakeflow Connect, SDP, Auto Loader, Spark, and Kafka.
• Design and implement CDC architectures using Debezium, Kafka/Kafka Connect, and relational databasessuch as PostgreSQL, MySQL, SQL Server, and Oracle.
• Implement streaming and event-driven data pipelines using Kafka, Spark Structured Streaming, and related technologies.
• Design and manage schema evolution and data contracts using Karapace / Schema Registry.
• Implement centralized governance using Unity Catalog, including catalogs, schemas, RBAC, row/column-level security, lineage, and data access policies.
• Develop metadata-driven ingestion frameworks, data quality, reconciliation, profiling, and observability solutions.
• Design Bronze, Silver, and Gold data layers and appropriate data modeling strategies for analytical workloads.
• Establish engineering best practices covering CI/CD, testing, deployment, monitoring, logging, and operational support.
• Use Databricks Asset Bundles (DAB) and CI/CD tools such as Jenkins/GitHub Actions for automated deployment.
• Work with cloud services such as AWS S3, IAM, networking, monitoring, and security services.
• Lead technical discussions with clients, translate business requirements into technical solutions, and drive architecture decisions.
• Troubleshoot complex data engineering, CDC, streaming, performance, and production issues.
• Mentor engineers and provide technical direction across data engineering initiatives. Must-Have Skills
• Strong hands-on experience with Databricks and Lakehouse architecture.
• Advanced Python and SQL skills.
• Strong expertise in Apache Spark / PySpark and distributed data processing.
• Hands-on experience with Unity Catalog and Delta Lake.
• Experience with Lakeflow Connect, SDP / Spark Declarative Pipelines, and Auto Loader.
• Strong understanding of CDC architectures using Debezium and Kafka.
• Hands-on experience with Kafka / Kafka Connect.
• Experience with Karapace or Schema Registry and schema evolution.
• Strong understanding of ETL/ELT, data modeling, data warehousing, streaming, and data integration patterns.
• Experience with production-grade data pipelines and orchestration.
• Strong understanding of cloud-native data services, particularly AWS.
• Experience with CI/CD and Databricks Asset Bundles (DAB).
• Experience leading technical implementations and working directly with business/client stakeholders.

Good to Have

• Experience with Snowflake and dbt.
• Experience with Apache Flink or other real-time processing frameworks.
• Experience implementing data governance, lineage, security, data quality, and observability.
• Experience with AWS S3, IAM, Glue, MSK/Kafka, and cloud networking.
• Experience with AI/ML data platforms and GenAI workloads.
• Databricks certifications, particularly Databricks Certified Data Engineer Professional.
• AWS Data Engineering/Data Analytics certifications. Other Expectations
• Strong ownership and problem-solving mindset.
• Ability to balance hands-on engineering with architecture and technical leadership.
• Strong client-facing and communication skills.
• Ability to mentor and guide engineering teams.
• Willingness to adapt to new technologies and client environments.
• Willingness to travel within India and internationally for short/medium-term client assignments.