Senior Data Engineer: 1B+ Record Scale (Azure Databricks)

TekWissen LLC; Bellevue (WA); On-site; USD 150,000 - 210,000; Full time

Job summary

TekWissen LLC is seeking an experienced Senior Data Engineer in Bellevue, WA to design and implement production-grade data solutions for Identity Resolution and large-scale customer data initiatives. The candidate will work onsite on a 7-month temporary assignment, delivering scalable pipelines and optimizing Spark performance.

The role requires strong Python and SQL skills, extensive experience with Azure Databricks, PySpark, and ADF, and the ability to drive solutions from design through

Qualifications

• 7+ years of Data Engineering experience.
• Proven experience working with very large-scale datasets, preferably 1B+ records.
• Strong hands-on experience with Azure Databricks and PySpark.
• Strong Python and SQL skills.
• Experience with Azure Data Factory (ADF) and Azure data services.
• Strong understanding of distributed processing and Spark performance optimization.
• Experience delivering production-grade, scalable data solutions.
• Ability to identify technical constraints early and make timely technical decisions.
• Strong problem-solving and communication skills.

Responsibilities

• Design and optimize large-scale data processing solutions capable of operating at 1B+ record scale.
• Develop scalable data pipelines using Azure Databricks, PySpark, Python, and ADF.
• Optimize Spark workloads including large-scale joins, partitioning, shuffling, data skew, memory, and compute utilization.
• Design and improve solutions for Identity Resolution, Entity Resolution, Record Linkage, Deduplication, or similar high-volume matching problems.
• Evaluate technical approaches against production data volumes and infrastructure constraints.
• Identify and communicate scalability and environment constraints early and recommend viable alternatives.
• Perform performance and scalability testing using production-like workloads.
• Build reliable data pipelines with appropriate validation, error handling, monitoring, and recovery mechanisms.
• Collaborate with architects, engineers, data scientists, and stakeholders to deliver solutions from design through production.

Skills

Python programming

SQL

Data engineering

Distributed data processing

Tools

Azure Databricks

PySpark

Azure Data Factory

Cosmos DB

Kafka

Job description

TekWissen LLC is seeking an experienced Senior Data Engineer in Bellevue, WA to design and implement production-grade data solutions for Identity Resolution and large-scale customer data initiatives. The candidate will work onsite on a 7-month temporary assignment, delivering scalable pipelines and optimizing Spark performance.

The role requires strong Python and SQL skills, extensive experience with Azure Databricks, PySpark, and ADF, and the ability to drive solutions from design through