Azure Databricks Technical Lead - PySpark Expert

Cognizant; Seattle (WA); Hybrid; USD 120,000 - 135,000; Full time

Benefits offered by this job

Medical/Dental/Vision/Life Insurance

Paid holidays plus Paid Time Off

401(k) plan and contributions

Long-term/Short-term Disability

Paid Parental Leave

Employee Stock Purchase Plan

Job summary

Cognizant is seeking a seasoned Technical Lead with hands-on PySpark, Python, and Azure Databricks to design scalable data solutions and guide delivery in a hybrid day-shift environment. You will collaborate with cross-functional teams to support airline and enterprise analytics with measurable business impact, mentoring engineers and enforcing engineering best practices.

Responsibilities include designing robust pipelines, optimizing Spark jobs, and implementing comprehensive testing and

Qualifications

• Extensive PySpark development experience with large-scale processing.
• Strong Python programming with modular design and cloud integration.
• Hands-on Azure Databricks expertise including job orchestration and secret management.
• Knowledge of data warehousing concepts, data modeling, ETL/ELT patterns.
• Experience with airline/travel industry data is a plus.
• Ability to collaborate in hybrid work environments and communicate clearly.
• Strong debugging and root-cause analysis for production incidents.

Responsibilities

• Design robust data engineering solutions using PySpark and Python for analytics.
• Develop reusable data pipelines on Azure Databricks for batch and near real-time processing.
• Translate analytical needs into technical designs aligned with enterprise data architecture.
• Oversee code reviews and enforce engineering best practices.
• Guide selection and configuration of Azure Databricks clusters and related Azure services.
• Implement comprehensive testing strategies (unit/integration/regression).
• Monitor production pipelines with dashboards and alerts to reduce downtime.
• Collaborate with data scientists, analysts, and domain experts to deliver curated data products.
• Improve development workflows with version control and deployment automation.
• Document designs, data models, and procedures for knowledge sharing.
• Coordinate with security/compliance to ensure governance and regulatory adherence.
• Optimize Spark jobs by tuning partitions and resources.
• Mentor junior engineers with feedback and modern data engineering practices.

Skills

PySpark

Python

Azure Databricks

Data modeling

ETL/ELT

Performance tuning

Mentoring

Tools

Azure Data Lake

Synapse

Git

Job description

Cognizant is seeking a seasoned Technical Lead with hands-on PySpark, Python, and Azure Databricks to design scalable data solutions and guide delivery in a hybrid day-shift environment. You will collaborate with cross-functional teams to support airline and enterprise analytics with measurable business impact, mentoring engineers and enforcing engineering best practices.

Responsibilities include designing robust pipelines, optimizing Spark jobs, and implementing comprehensive testing and