Data Engineer (Kafka)

Greymatter Innovationz

Estado de México

Presencial

MXN 637.000 - 763.000

Jornada completa

hace 30 horas
Sé de los primeros/as/es en solicitar esta vacante

Consigue una respuesta de este empleador

Tu currículum y tu carta de presentación, adaptados exactamente a lo que buscan contratar.

Supera los filtros ATS

Descripción de la vacante

Greymatter Innovationz seeks a Lead Kafka Integration Engineer to design and support enterprise data pipelines using Kafka, Databricks, and SQL Server-based MDM platforms.

You will architect real-time ingestion, develop Kafka consumer services, and ensure scalable, resilient data delivery with CI/CD and IaC practices.

Formación

  • Strong hands-on experience with Apache Kafka (consumers, consumer groups, offset management, DLQ, Schema Registry).
  • Expertise in Databricks, Spark, Delta Lake, and Medallion Architecture (Bronze/Silver/Gold).
  • Experience building real-time and batch data ingestion pipelines from Kafka to Databricks and SQL Server.
  • Strong knowledge of SQL Server, stored procedures, data modeling, MDM, and data reconciliation.
  • Experience with CI/CD pipelines, DevOps, IaC (Terraform/Bicep), and environment automation.
  • Expertise in monitoring, observability, performance tuning, and production support.
  • Experience with Azure cloud services and related data tools.
  • Experience with MDM platforms and healthcare data environments.
  • Knowledge of XML processing, backfeed services, and downstream integrations.
  • Experience with Kafka Connect, event-driven architecture, and streaming analytics.
  • Azure Data Ecosystem: ADLS Gen2, ADF, Synapse; Databricks on Azure using PySpark and Delta Lake.

Responsabilidades

  • Design and implement end-to-end data ingestion and processing from Kafka to Databricks and SQL Server.
  • Architect, develop, and support Kafka consumer services with offset management, DLQ, and security.
  • Set up DevOps & IaC for Kafka and Databricks with CI/CD pipelines and observability.
  • Build and optimize Kafka-to-SQL Server pipelines, data models, stored procedures, and data quality controls.
  • Lead architecture, development, troubleshooting, and governance across Kafka, Databricks, SQL Server, and cloud data platforms.
  • Batch and streaming ETL/ELT workflows using Azure Databricks, ADF, Informatica; real-time ingestion via Kafka.
  • Implement and maintain MDM workflows, data models, and match/merge logic for a single source of truth.

Conocimientos

Apache Kafka
Databricks
SQL Server
CI/CD
Terraform
Azure
MDM
Python/Scala
Kafka Connect
Event-Driven

Herramientas

Terraform
Bicep

Descripción del empleo

Please note: This is an independent contractor position with a maximum monthly compensation of USD $3,400. No additional benefits are included.

Job Description
  • We are seeking a Lead Kafka Integration Engineer to design, develop, and support enterprise-scale data integration solutions leveraging Kafka, Databricks, and SQL Server-based MDM platforms. The role will be responsible for architecting scalable and resilient real-time data ingestion pipelines, implementing Kafka consumer services, and delivering data through Databricks Bronze, Silver, and Gold layers.
  • Design and implement end-to-end data ingestion and processing solutions from Kafka into Databricks Medallion Architecture (Bronze/Silver/Gold) and SQL Server-based MDM platforms, ensuring scalability, resiliency, and high-performance data delivery.
  • Architect, develop, and support enterprise-grade Kafka consumer services including consumer groups, offset management, schema validation, retry mechanisms, dead-letter queues (DLQ), monitoring, ing, security, and automated recovery capabilities.
  • Establish DevOps and operational frameworks for Kafka and Databricks using CI/CD pipelines, Infrastructure as Code (IaC), environment automation, deployment standards, observability, logging, capacity planning, and production support.
  • Build and optimize Kafka-to-SQL Server ingestion pipelines, data models, stored procedures, reconciliation processes, and XML backfeed services supporting master data management, downstream integrations, and data quality controls.
  • Lead architecture, development, troubleshooting, performance tuning, and governance across Kafka, Databricks, SQL Server, and cloud-native data platforms while ensuring reliability, maintainability, security, and operational excellence.
  • Batch & Streaming Pipelines: Design, develop, and optimize scalable ETL/ELT workflows using Azure Databricks, Azure Data Factory, and Informatica, alongside real-time data ingestion pipelines using Apache Kafka.
  • Master Data Management (MDM): Implement and maintain MDM workflows, data models, and match/merge logic to establish a reliable single source of truth across enterprise systems.
  • Azure Lakehouse Management: Manage and optimize data storage, partitioning, and transformation layers within ADLS Gen2 and Delta Lake to ensure high performance and cost efficiency.
  • SQL & Performance Tuning: Write complex SQL queries, perform deep data analysis, and troubleshoot bottlenecks across streaming and batch workloads.
Mandatory Skills
  • Strong hands on experience with Apache Kafka (Consumers, Consumer Groups, Offset Management, DLQ, Schema Registry, Security)
  • Expertise in Databricks, Spark, Delta Lake, and Medallion Architecture (Bronze/Silver/Gold)
  • Experience building real-time and batch data ingestion pipelines from Kafka to Databricks and SQL Server
  • Strong knowledge of SQL Server, Stored Procedures, Data Modeling, MDM, and Data Reconciliation processes
  • Experience with CI/CD pipelines, DevOps practices, Infrastructure as Code (Terraform/Bicep), and environment automation
  • Expertise in monitoring, ing, observability, performance tuning, and production support
  • Experience with cloud platforms such as Azure and related data services
  • Experience with Master Data Management (MDM) platforms and healthcare data environments
  • Knowledge of XML processing, backfeed services, and downstream system integrations
  • Experience with Kafka Connect, Event-Driven Architecture, and Streaming Analytics
  • Azure Data Ecosystem: Hands-on experience building and managing cloud data solutions using Azure services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory (ADF), and Azure Synapse Analytics.
  • Databricks & Spark: Strong proficiency in Azure Databricks using PySpark, Spark SQL, and Delta Lake for scalable data processing and lakehouse architecture.
  • Real-Time Streaming (Kafka): Practical experience designing, building, and troubleshooting real-time event streaming pipelines using Apache Kafka (or Azure Event Hubs for Kafka).
  • Informatica & MDM: Proven experience with Informatica (IICS / PowerCenter) and Master Data Management (MDM) concepts, including data modeling, match/merge rules, and golden record creation.
  • Advanced SQL: Expert-level SQL skills for complex querying, data transformation, profiling, and performance tuning across relational and cloud data warehouses.'
Desired Skills
  • Exposure to containerization technologies such as Docker and Kubernetes
  • Familiarity with data governance, data quality frameworks, and security best practices
  • Strong leadership, stakeholder management, and mentoring capabilities
  • DevOps & Orchestration: Experience with CI/CD pipelines (Azure DevOps, GitHub Actions), Infrastructure as Code (Terraform/ARM), and workflow orchestration tools.
  • Programming & Scripting: Proficiency in Python or Scala beyond PySpark for automation, API integrations, and custom data processing.
  • Data Governance & Security: Familiarity with Unity Catalog, Microsoft Purview, RBAC security models, and data privacy compliance standards.
  • Certifications: Microsoft Certified: Azure Data Engineer Associate, Databricks Certified Data Engineer, or Informatica certifications.'
Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Data Engineer (Kafka)
Data Engineer (Kafka)

VeeRteq Solutions Inc. • Estado de México

Presencial
MXN 1.200.000 - 2.100.000
IT Data Engineer
IT Data Engineer

Eastman Kodak Company • Región Centro

Presencial
MXN 1.583.000 - 2.607.000
Azure Data Engineer
Azure Data Engineer

Vaiticka Solution • Estado de México

Presencial
MXN 1.660.000 - 2.397.000
Data Engineer With Databricks
Data Engineer With Databricks

VeeRteq Solutions Inc. • Estado de México

Presencial
MXN 350.000 - 750.000
Azure Data Engineer
Azure Data Engineer

Lingaro • México

Presencial
MXN 700.000 - 1.100.000
Senior Azure Databricks and PySpark
Senior Azure Databricks and PySpark

Daikin Manufacturing México • Monterrey

Híbrido
MXN 900.000 - 1.400.000
Data Engineer
Data Engineer

Pyramid Consulting, Inc • Aguascalientes

Híbrido
MXN 1.200.000 - 1.800.000
Cloud Engineer Azure Databricks
Cloud Engineer Azure Databricks

Turtle Trax S.A. • Región Centro

Presencial
MXN 1.000.000 - 1.500.000
Data Solution Architect
Data Solution Architect

Turtle Trax S.A. • Región Centro

A distancia
MXN 900.000 - 1.500.000
Data Engineer Mexico
Data Engineer Mexico

V4C.ai • México

A distancia
MXN 420.000 - 660.000
Health insurance
Growth opportunities
Parental leave
+1