Sr Data engineer

Cognizant; United States; On-site; USD 150,000 - 210,000; Full time

Job summary

Cognizant seeks an experienced Lead Data Engineer to architect and implement scalable data platforms in the hotel and hospitality domain. You will lead end-to-end data engineering efforts, drive best practices, and collaborate with stakeholders to translate analytics requirements into robust technical solutions.

The role emphasizes data quality, governance, and cost-efficient Databricks and Spark workloads across enterprise data products.

Qualifications

• Strong hands-on experience with Azure Databricks and Spark ecosystems.
• Expertise in PySpark development and distributed data processing.
• Proven ability to design and maintain large-scale data pipelines.
• Experience with data modeling, data integration, ETL/ELT processes.
• Knowledge of data quality, monitoring, and validation frameworks.

Responsibilities

• Lead the end-to-end design and implementation of scalable data engineering solutions using Azure Databricks, Spark, and PySpark.
• Design, develop, and optimize robust data pipelines and processing workloads for performance, reliability, and cost efficiency.
• Provide technical direction on PySpark/Spark development standards, data modeling, and architecture patterns.

Skills

Azure Databricks

Apache Spark

PySpark

Large-scale data pipelines

Data modeling

Data integration

ETL/ELT

Data governance

Data quality

Performance tuning

Cluster optimization

Mentoring

Stakeholder management

Translate requirements

Cloud architecture

Job description

Must have - 10+ years of experience to be considered.

**No visa sponsorship / transfer / c2c available for this role**

Role Overview

We are looking for an experienced Lead Data Engineer to lead the end-to-end design and implementation of scalable data engineering solutions using Azure Databricks, Apache Spark, and PySpark. The ideal candidate will provide technical leadership, drive engineering best practices, and build high-performance data platforms supporting analytics and business-critical use cases within the hotel and hospitality domain.

Key Responsibilities

• Lead the end-to-end design and implementation of scalable data engineering solutions using Azure Databricks, Spark, and PySpark.
• Design, develop, and optimize robust data pipelines and processing workloads while ensuring performance, reliability, scalability, and cost efficiency.
• Provide technical direction on PySpark/Spark development standards, data modeling, architecture patterns, and environment configuration.
• Integrate diverse data sources into unified, trusted datasets that support reporting, advanced analytics, and enterprise data products.
• Partner with product owners and business stakeholders to translate analytical and reporting requirements into clear technical solutions and implementation plans.
• Implement strong data quality, monitoring, validation, and error-handling frameworks to ensure accurate and reliable downstream data.
• Optimize Databricks clusters, resource utilization, job scheduling, and compute workloads to improve performance and manage cloud costs.
• Build and promote reusable components, shared libraries, frameworks, and configuration templates to accelerate engineering delivery.
• Collaborate with architecture, security, and operations teams to ensure solutions comply with data governance, privacy, security, and enterprise cloud standards.

.

Required Skills

• Strong hands-on experience with Azure Databricks
• Advanced expertise in Apache Spark and PySpark
• Strong experience designing and developing large-scale data pipelines
• Experience with data modeling, data integration, ETL/ELT, and distributed data processing
• Expertise in Databricks performance tuning, cluster optimization, and workload management
• Experience implementing data quality, monitoring, and validation frameworks
• Strong understanding of cloud data architecture, security, governance, and privacy
• Experience developing reusable data engineering frameworks and libraries
• Strong technical leadership, mentoring, and stakeholder management skills
• Ability to translate complex business requirements into scalable technical solutions

Preferred

• Experience working with hotel, hospitality, booking, customer behavior, or operational data
• Experience leading enterprise-scale Azure Databricks implementations
• Experience working in large, distributed or global engineering environments

Core Tech Stack:

Azure Databricks | Apache Spark | PySpark | Azure | Data Engineering | ETL/ELT | Data Modeling | Data Quality | Performance Optimization