Coffeebeans.io - Senior Data Engineer - Python

CoffeeBeans; Bengaluru Urban; Hybrid; INR 1,500,000 - 2,800,000; Full time

Job summary

CoffeeBeans Consulting in Bangalore is seeking a Data Engineer L2 to design and implement scalable data pipelines and Lakehouse architectures. The role emphasizes building batch and real-time ingestion with Spark, Kafka, and Delta Lake in a hybrid work setup.

Ideal candidates have 4–7 years of data engineering experience, strong Python/SQL skills, and hands-on experience with Databricks, Unity Catalog, and AWS services. You will collaborate with clients and mentor engineers on complex data tasks.

Qualifications

• 4-7 years of experience in data engineering.
• Experience designing Databricks Lakehouse architectures.
• Strong Python and SQL skills.

Responsibilities

• Design and implement enterprise-grade Databricks Lakehouse architectures using Delta Lake and Unity Catalog.
• Build scalable batch and real-time data ingestion pipelines using Lakeflow Connect, SDP, Auto Loader, Spark, and Kafka.
• Design and implement CDC architectures using Debezium, Kafka/Kafka Connect, and relational databases.
• Implement streaming pipelines using Kafka and Spark Structured Streaming.
• Governance using Unity Catalog with RBAC and data access policies.
• Develop metadata-driven ingestion, data quality, and observability solutions.
• Design data layers (Bronze/Silver/Gold) and data modeling strategies for analytics.
• Establish CI/CD, testing, deployment, monitoring, and logging practices.
• Use Databricks Asset Bundles (DAB) and CI/CD tools for deployment.
• Work with AWS services (S3, IAM, networking, monitoring, security).
• Lead technical discussions with clients and translate requirements into solutions.

Skills

Databricks Lakehouse

Python

SQL

Spark / PySpark

Unity Catalog

Delta Lake

Lakeflow Connect

SDP

Auto Loader

Debezium

Kafka

Kafka Connect

Schema Registry

ETL/ELT

Data modeling

Data warehousing

Streaming

CI/CD

Databricks Asset Bundles

Cloud - AWS

Leadership

Tools

Databricks

Lakehouse Architecture

Unity Catalog

Delta Lake

Kafka

Kubernetes

Jenkins / GitHub Actions

S3

IAM

Job description

Data Engineer L2

Experience: 3 - 10 years in data engineering.

Data Engineer L2

Experience: 3 - 10 years in data engineering.

Location: Bangalore.

Work Mode: Bangalore - Hybrid.

Role Overview

Join CoffeeBeans Consulting as a Data Engineer L2 and immerse yourself in a transformative role where your expertise will directly contribute to the future of AI. Located in Bangalore, this position offers a unique opportunity to work at the forefront of data engineering, shaping the way businesses leverage their data to drive innovation. With 4 - 7 years of experience, you will play a pivotal role in building and optimizing scalable data pipelines that empower analytics and AI/ML solutions. This is not just a job; it's a chance to elevate your career in a company that values engineering excellence and client impact.

Key Responsibilities

• Design and implement enterprise-grade Databricks Lakehouse architectures using Delta Lake and Unity Catalog.
• Build scalable batch and real-time data ingestion pipelines using Lakeflow Connect, SDP, Auto Loader, Spark, and Kafka.
• Design and implement CDC architectures using Debezium, Kafka/Kafka Connect, and relational databases such as PostgreSQL, MySQL, SQL Server, and Oracle.
• Implement streaming and event-driven data pipelines using Kafka, Spark Structured Streaming, and related technologies.
• Design and manage schema evolution and data contracts using Karapace / Schema Registry.
• Implement centralized governance using Unity Catalog, including catalogs, schemas, RBAC, row/column-level security, lineage, and data access policies.
• Develop metadata-driven ingestion frameworks, data quality, reconciliation, profiling, and observability solutions.
• Design Bronze, Silver, and Gold data layers and appropriate data modeling strategies for analytical workloads.
• Establish engineering best practices covering CI/CD, testing, deployment, monitoring, logging, and operational support.
• Use Databricks Asset Bundles (DAB) and CI/CD tools such as Jenkins/GitHub Actions for automated deployment.
• Work with cloud services such as AWS S3, IAM, networking, monitoring, and security services.
• Lead technical discussions with clients, translate business requirements into technical solutions, and drive architecture decisions.
• Troubleshoot complex data engineering, CDC, streaming, performance, and production issues.
• Mentor engineers and provide technical direction across data engineering initiatives.

Must-Have Skills

• Strong hands‑on experience with Databricks and Lakehouse architecture.
• Advanced Python and SQL skills.
• Strong expertise in Apache Spark / PySpark and distributed data processing.
• Hands‑on experience with Unity Catalog and Delta Lake.
• Experience with Lakeflow Connect, SDP / Spark Declarative Pipelines, and Auto Loader.
• Strong understanding of CDC architectures using Debezium and Kafka.
• Hands‑on experience with Kafka / Kafka Connect.
• Experience with Karapace or Schema Registry and schema evolution.
• Strong understanding of ETL/ELT, data modeling, data warehousing, streaming, and data integration patterns.
• Experience with production‑grade data pipelines and orchestration.
• Strong understanding of cloud‑native data services, particularly AWS.
• Experience with CI/CD and Databricks Asset Bundles (DAB).
• Experience leading technical implementations and working directly with business/client stakeholders.

Good To Have

• Experience with Snowflake and dbt.
• Experience with Apache Flink or other real‑time processing frameworks.
• Experience implementing data governance, lineage, security, data quality, and observability.
• Experience with AWS S3, IAM, Glue, MSK/Kafka, and cloud networking.
• Experience designing metadata‑driven data platforms.
• Experience with AI/ML data platforms and GenAI workloads.
• Databricks certifications, particularly Databricks Certified Data Engineer Professional.
• AWS Data Engineering/Data Analytics certifications.

Other Expectations

• Strong ownership and problem‑solving mindset.
• Ability to balance hands‑on engineering with architecture and technical leadership.
• Strong client‑facing and communication skills.
• Ability to mentor and guide engineering teams.
• Willingness to adapt to new technologies and client environments.
• Willingness to travel within India and internationally for short/medium‑term client assignments.

(ref:hirist.tech)