Data Engineer

CoffeeBeans Consulting LLP; Bengaluru; Presencial; INR 2.000.000 - 3.200.000; Jornada completa

Descripción de la vacante

CoffeeBeans Consulting LLP in Bangalore is hiring a Data Engineer L3 with 7–10 years of experience to design and implement scalable data platforms for analytics and AI workloads. You will architect Lakehouse solutions, build streaming and batch pipelines, and ensure governance across data products in a client-facing role.

You will lead architecture decisions, mentor engineers, and collaborate with stakeholders to deliver high-impact data-driven outcomes in an on-site setup.

Formación

• 7–10 years of data engineering experience in large-scale data platforms.
• Expertise with Databricks Lakehouse architecture and Delta Lake.
• Strong Python and SQL with deep Spark experience.

Responsabilidades

• Design and implement Databricks Lakehouse architectures using Delta Lake and Unity Catalog.
• Build scalable batch and real-time data pipelines using Lakeflow Connect, SDP, Auto Loader, Spark, and Kafka.
• Design and implement CDC architectures using Debezium, Kafka Connect, and relational databases (PostgreSQL, MySQL, SQL Server, Oracle).
• Implement streaming and event-driven data pipelines using Kafka, Spark Structured Streaming, and related technologies.
• Design and manage schema evolution and data contracts using Karapace / Schema Registry.
• Implement centralized governance using Unity Catalog, including RBAC, security, lineage, and data access policies.
• Develop metadata-driven ingestion frameworks, data quality, reconciliation, profiling, and observability solutions.
• Design Bronze, Silver, and Gold data layers and data modeling strategies for analytical workloads.
• Establish CI/CD, testing, deployment, monitoring, logging, and operational support.
• Use Databricks Asset Bundles (DAB) and CI/CD tools like Jenkins/GitHub Actions for automated deployment.
• Work with cloud services such as AWS S3, IAM, networking, monitoring, and security services.
• Lead technical discussions with clients, translate requirements into technical solutions, and drive architecture decisions.
• Troubleshoot complex data engineering, CDC, streaming, performance, and production issues.
• Mentor engineers and provide technical direction across data engineering initiatives.

Conocimientos

Snowflake

Lakehouse

Python

SQL

Spark/PySpark

Unity Catalog

Delta Lake

Lakeflow Connect

SDP

Auto Loader

Debezium

Kafka

Kafka Connect

Schema Registry

Karapace

ETL/ELT

Data Warehousing

CI/CD

Databricks

AWS

Leadership

Herramientas

Databricks

Lakeflow Connect

Kafka

Spark

Jenkins

GitHub Actions

Unity Catalog

Delta Lake

Descripción del empleo

Data Engineer L3

Experience: 7-10 years | Location: Bangalore. | Work Mode: On-site.

About Us

CoffeeBeans Consulting is a tech consulting firm focused on making organizations AI-ready by structuring their data efficiently across various sources and enabling AI-driven solutions. We specialize in data architecture, pipelines, governance, MLOps, and Gen AI solutions.

Role Overview

Join CoffeeBeans Consulting as a Data Engineer L3 and immerse yourself in a transformative role where your expertise will directly contribute to the future of AI. Located in Bangalore, this position offers a unique opportunity to work at the forefront of data engineering, shaping the way businesses leverage their data to drive innovation. With 7-10years of experience, you will play a pivotal role in building and optimizing scalable data pipelines that empower analytics and AI/ML solutions. This is not just a job; its a chance to elevate your career in a company that values engineering excellence and client impact.

Key Responsibilities

• Design and implement enterprise-grade Databricks Lakehouse architectures using Delta Lake and Unity Catalog.
• Build scalable batch and real-time data ingestion pipelines using Lakeflow Connect, SDP, Auto Loader, Spark, and Kafka.
• Design and implement CDC architectures using Debezium, Kafka/Kafka Connect, and relational databasessuch as PostgreSQL, MySQL, SQL Server, and Oracle.
• Implement streaming and event-driven data pipelines using Kafka, Spark Structured Streaming, and related technologies.
• Design and manage schema evolution and data contracts using Karapace / Schema Registry.
• Implement centralized governance using Unity Catalog, including catalogs, schemas, RBAC, row/column-level security, lineage, and data access policies.
• Develop metadata-driven ingestion frameworks, data quality, reconciliation, profiling, and observability solutions.
• Design Bronze, Silver, and Gold data layers and appropriate data modeling strategies for analytical workloads.
• Establish engineering best practices covering CI/CD, testing, deployment, monitoring, logging, and operational support.
• Use Databricks Asset Bundles (DAB) and CI/CD tools such as Jenkins/GitHub Actions for automated deployment.
• Work with cloud services such as AWS S3, IAM, networking, monitoring, and security services.
• Lead technical discussions with clients, translate business requirements into technical solutions, and drive architecture decisions.
• Troubleshoot complex data engineering, CDC, streaming, performance, and production issues.
• Mentor engineers and provide technical direction across data engineering initiatives.
• Must-Have Skills
• Strong hands-on experience with Snowflake and Lakehouse architecture.
• Advanced Python and SQL skills.
• Strong expertise in Apache Spark / PySpark and distributed data processing.
• Hands-on experience with Unity Catalog and Delta Lake.
• Experience with Lakeflow Connect, SDP / Spark Declarative Pipelines, and Auto Loader.
• Strong understanding of CDC architectures using Debezium and Kafka.
• Hands-on experience with Kafka / Kafka Connect.
• Experience with Karapace or Schema Registry and schema evolution.
• Strong understanding of ETL/ELT, data modeling, data warehousing, streaming, and data integration patterns.
• Experience with production-grade data pipelines and orchestration.
• Strong understanding of cloud-native data services, particularly AWS.
• Experience with CI/CD and Databricks Asset Bundles (DAB).
• Experience leading technical implementations and working directly with business/client stakeholders.

Good to Have

• Experience with Snowflake and dbt.
• Experience with Apache Flink or other real-time processing frameworks.
• Experience implementing data governance, lineage, security, data quality, and observability.
• Experience with AWS S3, IAM, Glue, MSK/Kafka, and cloud networking.
• Experience designing metadata-driven data platforms.
• Experience with AI/ML data platforms and GenAI workloads.
• Databricks certifications, particularly Databricks Certified Data Engineer Professional.
• AWS Data Engineering/Data Analytics certifications. Other Expectations
• Strong ownership and problem-solving mindset.
• Ability to balance hands-on engineering with architecture and technical leadership.
• Strong client-facing and communication skills.
• Ability to mentor and guide engineering teams.
• Willingness to adapt to new technologies and client environments.
• Willingness to travel within India and internationally for short/medium-term client assignments.