Lead Kafka & Databricks Data Engineer (Azure, MDM)

VeeRteq Solutions Inc.

Estado de México

On-site

MXN 1,200,000 - 2,100,000

Full time

29 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

VeeRteq Solutions Inc. is seeking a Lead Kafka Integration Engineer to design, develop, and support enterprise-scale data integration solutions leveraging Kafka, Databricks, and SQL Server-based MDM platforms.

You will architect scalable, real-time data ingestion pipelines and implement Kafka consumer services within a Databricks Medallion Architecture. You will also establish DevOps practices with CI/CD, IaC, and monitoring, while leading governance and performance tuning across Azure data

Qualifications

  • Strong hands-on experience with Apache Kafka (Consumers, Consumer Groups, Offset Management, DLQ, Schema Registry, Security).
  • Expertise in Databricks, Spark, Delta Lake, and Medallion Architecture (Bronze/Silver/Gold).
  • Experience building real-time and batch data ingestion pipelines from Kafka to Databricks and SQL Server.
  • Strong knowledge of SQL Server, Stored Procedures, Data Modeling, MDM, and Data Reconciliation processes.
  • Experience with CI/CD pipelines, DevOps practices, Infrastructure as Code (Terraform/Bicep), and environment automation.
  • Expertise in monitoring, observability, performance tuning, and production support.
  • Experience with cloud platforms such as Azure and related data services.
  • Experience with Master Data Management (MDM) platforms and healthcare data environments.
  • Knowledge of XML processing, backfeed services, and downstream system integrations.
  • Experience with Kafka Connect, Event-Driven Architecture, and Streaming Analytics Azure Data Ecosystem: Hands-on experience building and managing cloud data solutions using Azure services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory (ADF), and Azure Synapse Analytics.
  • Databricks & Spark: Strong proficiency in Azure Databricks using PySpark, Spark SQL, and Delta Lake for scalable data processing and lakehouse architecture.
  • Real-Time Streaming (Kafka): Practical experience designing, building, and troubleshooting real-time event streaming pipelines using Apache Kafka (or Azure Event Hubs for Kafka).
  • Informatica & MDM: Proven experience with Informatica (IICS / PowerCenter) and Master Data Management (MDM) concepts, including data modeling, match/merge rules, and golden record creation.
  • Advanced SQL: Expert-level SQL skills for complex querying, data transformation, profiling, and performance tuning across relational and cloud data warehouses.

Responsibilities

  • Design and implement end-to-end data ingestion and processing solutions from Kafka into Databricks Medallion Architecture (Bronze/Silver/Gold) and SQL Server-based MDM platforms.
  • Architect, develop, and support enterprise-grade Kafka consumer services including consumer groups, offset management, schema validation, retry mechanisms, dead-letter queues (DLQ), monitoring, ing, security, and automated recovery capabilities.
  • Establish DevOps and operational frameworks for Kafka and Databricks using CI/CD pipelines, Infrastructure as Code (IaC), environment automation, deployment standards, observability, logging, capacity planning, and production support.
  • Build and optimize Kafka-to-SQL Server ingestion pipelines, data models, stored procedures, reconciliation processes, and XML backfeed services supporting master data management, downstream integrations, and data quality controls.
  • Lead architecture, development, troubleshooting, performance tuning, and governance across Kafka, Databricks, SQL Server, and cloud-native data platforms while ensuring reliability, maintainability, security, and operational excellence.
  • Batch & Streaming Pipelines: Design, develop, and optimize scalable ETL/ELT workflows using Azure Databricks, Azure Data Factory, and Informatica, alongside real-time data ingestion pipelines using Apache Kafka.
  • Master Data Management (MDM): Implement and maintain MDM workflows, data models, and match/merge logic to establish a reliable single source of truth across enterprise systems.
  • Azure Lakehouse Management: Manage and optimize data storage, partitioning, and transformation layers within ADLS Gen2 and Delta Lake to ensure high performance and cost efficiency.
  • SQL & Performance Tuning: Write complex SQL queries, perform deep data analysis, and troubleshoot bottlenecks across streaming and batch workloads.

Skills

Kafka mastery
Databricks
SQL Server
MDM
CI/CD
IaC
Azure cloud
OBS/Monitoring
Security
DLQ handling
Schema registry
Spark SQL
Delta Lake
Python/Scala
XML processing

Tools

Kafka Connect
Azure Data Factory
ADF
Terraform
Bicep
ARM templates
GitHub Actions
Azure Synapse
Informatica
SQL stored procedures

Job description

VeeRteq Solutions Inc. is seeking a Lead Kafka Integration Engineer to design, develop, and support enterprise-scale data integration solutions leveraging Kafka, Databricks, and SQL Server-based MDM platforms.

You will architect scalable, real-time data ingestion pipelines and implement Kafka consumer services within a Databricks Medallion Architecture. You will also establish DevOps practices with CI/CD, IaC, and monitoring, while leading governance and performance tuning across Azure data

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer (Kafka)
Data Engineer (Kafka)

VeeRteq Solutions Inc. • Estado de México

On-site
MXN 1,200,000 - 2,100,000
Databricks Data Engineer for Healthcare Pipelines
Databricks Data Engineer for Healthcare Pipelines

VeeRteq Solutions Inc. • Estado de México

On-site
MXN 350,000 - 750,000
Senior Data Engineer — Databricks & Azure, Global Impact
Senior Data Engineer — Databricks & Azure, Global Impact

KMS Technology, Inc. • Región Centro

On-site
MXN 900,000 - 1,400,000
15 days PTO
Major Medical Insurance
Annual performance bonus
+6
Lead Data Engineer: Databricks & Ingestion Architect
Lead Data Engineer: Databricks & Ingestion Architect

Pyramid Consulting, Inc • Aguascalientes

On-site
MXN 900,000 - 1,500,000
Senior Data Engineer: Azure Databricks & Spark Specialist
Senior Data Engineer: Azure Databricks & Spark Specialist

Applaudo • Ciudad de México

On-site
MXN 900,000 - 1,400,000
IT Data Engineer
IT Data Engineer

Eastman Kodak Company • Región Centro

On-site
MXN 1,583,000 - 2,607,000
Data Engineer
Data Engineer

Pyramid Consulting, Inc • Aguascalientes

On-site
MXN 900,000 - 1,500,000
Senior Data Engineer: Scalable Pipelines on Azure Databricks
Senior Data Engineer: Scalable Pipelines on Azure Databricks

Applaudo • Mexico

On-site
MXN 900,000 - 1,200,000
Senior Data Engineer: Azure Databricks & PySpark
Senior Data Engineer: Azure Databricks & PySpark

EPAM Systems • Mexico

Remote
MXN 900,000 - 1,500,000
Healthcare benefits
Paid time off
Upskilling and certification courses
+1
ETL QA Specialist: Data Integrity & MDM Validation
ETL QA Specialist: Data Integrity & MDM Validation

VeeRteq Solutions Inc. • Estado de México

On-site
MXN 420,000 - 640,000