Senior Data Engineer

Coffeebeans Consulting; Bengaluru; On-site; INR 3,000,000 - 6,000,000; Full time

Job summary

CoffeeBeans Consulting is seeking a Data Engineer in Bangalore to build and optimize scalable data pipelines and AI-ready data platforms. You will design Lakehouse architectures, implement streaming and CDC pipelines, and drive architectural decisions.

This role requires 6–10 years of data engineering experience and deep expertise in Databricks, Spark, Python, and SQL. Join a team focused on data governance, observability, and cloud-native solutions.

Qualifications

• 6–10 years of experience in data engineering.
• Expertise with Databricks and Lakehouse architecture.
• Strong Python and SQL skills; deep knowledge of Spark/PySpark.

Responsibilities

• Design enterprise-grade data lakehouse architectures using Databricks Delta Lake and Unity Catalog.
• Build scalable batch and real-time data pipelines with Lakeflow, SDP, Auto Loader, Kafka.
• Implement CDC, streaming, and data contracts using Debezium, Kafka, and Schema Registry.
• Establish governance, data modeling, and data quality across analytics layers.
• Mentor engineers and lead technical discussions with clients.

Skills

Python

SQL

Apache Spark / PySpark

Databricks Lakehouse

Unity Catalog

Tools

Lakeflow Connect

Kafka / Kafka Connect

Delta Lake

Auto Loader

SDP / Spark Declarative Pipelines

Karapace / Schema Registry

Debezium

Jenkins / GitHub Actions

AWS

Job description

Data Engineer

Experience: 6-10 years in data engineering. | Location: Bangalore. | Work Mode: On-site.

About Us

CoffeeBeans Consulting is a tech consulting firm focused on making organizations AI-ready by structuring their data efficiently across various sources and enabling AI-driven solutions. We specialize in data architecture, pipelines, governance, MLOps, and Gen AI solutions.

Role Overview

Join CoffeeBeans Consulting as a Data Engineer and immerse yourself in a transformative role where your expertise will directly contribute to the future of AI. Located in Bangalore, this position offers a unique opportunity to work at the forefront of data engineering, shaping the way businesses leverage their data to drive innovation. With 6-10 years of experience, you will play a pivotal role in building and optimizing scalable data pipelines that empower analytics and AI/ML solutions. This is not just a job; its a chance to elevate your career in a company that values engineering excellence and client impact.

Key Responsibilities

• Design and implement enterprise-grade Databricks Lakehouse architectures using Delta Lake and Unity Catalog.
• Build scalable batch and real-time data ingestion pipelines using Lakeflow Connect, SDP, Auto Loader, Spark, and Kafka.
• Design and implement CDC architectures using Debezium, Kafka/Kafka Connect, and relational databasessuch as PostgreSQL, MySQL, SQL Server, and Oracle.
• Implement streaming and event-driven data pipelines using Kafka, Spark Structured Streaming, and related technologies.
• Design and manage schema evolution and data contracts using Karapace / Schema Registry.
• Implement centralized governance using Unity Catalog, including catalogs, schemas, RBAC, row/column-level security, lineage, and data access policies.
• Develop metadata-driven ingestion frameworks, data quality, reconciliation, profiling, and observability solutions.
• Design Bronze, Silver, and Gold data layers and appropriate data modeling strategies for analytical workloads.
• Establish engineering best practices covering CI/CD, testing, deployment, monitoring, logging, and operational support.
• Use Databricks Asset Bundles (DAB) and CI/CD tools such as Jenkins/GitHub Actions for automated deployment.
• Work with cloud services such as AWS S3, IAM, networking, monitoring, and security services.
• Lead technical discussions with clients, translate business requirements into technical solutions, and drive architecture decisions.
• Troubleshoot complex data engineering, CDC, streaming, performance, and production issues.
• Mentor engineers and provide technical direction across data engineering initiatives. Must-Have Skills
• Strong hands-on experience with Databricks and Lakehouse architecture.
• Advanced Python and SQL skills.
• Strong expertise in Apache Spark / PySpark and distributed data processing.
• Hands-on experience with Unity Catalog and Delta Lake.
• Experience with Lakeflow Connect, SDP / Spark Declarative Pipelines, and Auto Loader.
• Strong understanding of CDC architectures using Debezium and Kafka.
• Hands-on experience with Kafka / Kafka Connect.
• Experience with Karapace or Schema Registry and schema evolution.
• Strong understanding of ETL/ELT, data modeling, data warehousing, streaming, and data integration patterns.
• Experience with production-grade data pipelines and orchestration.
• Strong understanding of cloud-native data services, particularly AWS.
• Experience with CI/CD and Databricks Asset Bundles (DAB).
• Experience leading technical implementations and working directly with business/client stakeholders.

Good to Have

• Experience with Snowflake and dbt.
• Experience with Apache Flink or other real-time processing frameworks.
• Experience implementing data governance, lineage, security, data quality, and observability.
• Experience with AWS S3, IAM, Glue, MSK/Kafka, and cloud networking.
• Experience designing metadata-driven data platforms.
• Experience with AI/ML data platforms and GenAI workloads.
• Databricks certifications, particularly Databricks Certified Data Engineer Professional.
• AWS Data Engineering/Data Analytics certifications. Other Expectations
• Strong ownership and problem-solving mindset.
• Ability to balance hands-on engineering with architecture and technical leadership.
• Strong client-facing and communication skills.
• Ability to mentor and guide engineering teams.
• Willingness to adapt to new technologies and client environments.
• Willingness to travel within India and internationally for short/medium-term client assignments.