Senior Data Engineer

CoffeeBeans; Bengaluru; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

CoffeeBeans is seeking seasoned data engineers to own end-to-end data engagements, from architecture to delivery. You will lead initiatives and guide teams in building scalable data platforms that support BI, analytics, and AI/ML.

Key work includes designing Databricks Lakehouse architectures with Delta Lake, building ingestion pipelines with Kafka and Spark, and implementing robust data governance and data modeling for analytical workloads.

Qualifications

• Must have hands-on Databricks and Lakehouse architecture experience.
• Advanced Python and SQL skills.
• Strong Apache Spark / PySpark expertise and distributed data processing.
• Hands-on with Unity Catalog and Delta Lake.
• Experience with Lakeflow Connect, SDP / Spark Declarative Pipelines, and Auto Loader.
• Knowledge of CDC architectures using Debezium and Kafka.

Responsibilities

• Design and implement Databricks Lakehouse architectures with Delta Lake and Unity Catalog.
• Build scalable batch and real-time data ingestion pipelines using Lakeflow Connect, SDP, Auto Loader, Spark, and Kafka.
• Design CDC architectures using Debezium, Kafka/Kafka Connect, and relational databases like PostgreSQL, MySQL, SQL Server, Oracle.

Skills

Databricks Lakehouse

Python

SQL

Apache Spark/PySpark

Kafka

Delta Lake

Unity Catalog

Lakeflow Connect

SDP

Auto Loader

Debezium

Kafka Connect

Schema Registry

Karapace

CI/CD (Jenkins/GitHub Actions)

Tools

Databricks

Jenkins

GitHub Actions

Lakeflow Connect

SDP

Auto Loader

Kafka

Debezium

Kafka Connect

Unity Catalog

Delta Lake

Karapace

Schema Registry

Job description

We’re looking for seasoned data engineers who can take end-to-end ownership of complex data engineering engagements—right from architecture and design to development and delivery. You’ll work with client stakeholders, lead data initiatives, and guide teams in building modern, scalable data platforms that support business intelligence, advanced analytics, and AI/ML applications.

Key Responsibilities

• Design and implement enterprise-grade Databricks Lakehouse architectures using Delta Lake and Unity Catalog.
• Build scalable batch and real-time data ingestion pipelines using Lakeflow Connect, SDP, Auto Loader, Spark, and Kafka.
• Design and implement CDC architectures using Debezium, Kafka/Kafka Connect, and relational databasessuch as PostgreSQL, MySQL, SQL Server, and Oracle.
• Implement streaming and event-driven data pipelines using Kafka, Spark Structured Streaming, and related technologies.
• Design and manage schema evolution and data contracts using Karapace / Schema Registry.
• Implement centralized governance using Unity Catalog, including catalogs, schemas, RBAC, row/column-level security, lineage, and data access policies.
• Develop metadata-driven ingestion frameworks, data quality, reconciliation, profiling, and observability solutions.
• Design Bronze, Silver, and Gold data layers and appropriate data modeling strategies for analytical workloads.
• Establish engineering best practices covering CI/CD, testing, deployment, monitoring, logging, and operational support.
• Use Databricks Asset Bundles (DAB) and CI/CD tools such as Jenkins/GitHub Actions for automated deployment.
• Work with cloud services such as AWS S3, IAM, networking, monitoring, and security services.
• Lead technical discussions with clients, translate business requirements into technical solutions, and drive architecture decisions.
• Troubleshoot complex data engineering, CDC, streaming, performance, and production issues.
• Mentor engineers and provide technical direction across data engineering initiatives. Must-Have Skills
• Strong hands-on experience with Databricks and Lakehouse architecture.
• Advanced Python and SQL skills.
• Strong expertise in Apache Spark / PySpark and distributed data processing.
• Hands-on experience with Unity Catalog and Delta Lake.
• Experience with Lakeflow Connect, SDP / Spark Declarative Pipelines, and Auto Loader.
• Strong understanding of CDC architectures using Debezium and Kafka.
• Hands-on experience with Kafka / Kafka Connect.
• Experience with Karapace or Schema Registry and schema evolution.
• Strong understanding of ETL/ELT, data modeling, data warehousing, streaming, and data integration patterns.
• Experience with production-grade data pipelines and orchestration.
• Strong understanding of cloud-native data services, particularly AWS.
• Experience with CI/CD and Databricks Asset Bundles (DAB).
• Experience leading technical implementations and working directly with business/client stakeholders.

Good to Have

• Experience with Snowflake and dbt.
• Experience with Apache Flink or other real-time processing frameworks.
• Experience implementing data governance, lineage, security, data quality, and observability.
• Experience with AWS S3, IAM, Glue, MSK/Kafka, and cloud networking.
• Experience with AI/ML data platforms and GenAI workloads.
• Databricks certifications, particularly Databricks Certified Data Engineer Professional.
• AWS Data Engineering/Data Analytics certifications. Other Expectations
• Strong ownership and problem-solving mindset.
• Ability to balance hands-on engineering with architecture and technical leadership.
• Strong client-facing and communication skills.
• Ability to mentor and guide engineering teams.
• Willingness to adapt to new technologies and client environments.
• Willingness to travel within India and internationally for short/medium-term client assignments.