Own enterprise-grade lakehouse data engineering on Azure, building and optimizing scalable ETL/ELT pipelines with Azure Databricks.
Responsibilities
- Design and build scalable ETL/ELT pipelines using Azure Databricks with PySpark and Spark SQL
- Develop ingestion workflows from multiple source systems into Bronze/Silver/Gold layers (e.g., Event Hub, APIs, Azure SQL, Blob Storage)
- Apply best practices for Delta Lake and Unity Catalog, including schema evolution, versioning, and governance
- Optimize pipeline performance and cost using partitioning, caching, clustering, and Spark tuning
- Integrate Databricks with key Azure services including ADLS Gen2, Azure Data Factory (ADF), Azure Synapse, Event Hub, Key Vault, and Azure DevOps
- Implement secure access controls using managed identities, RBAC, and data protection practices such as masking
- Collaborate with analytics and business teams to deliver trusted datasets, metrics, and KPIs
- Provide technical leadership through mentoring, code reviews, and architectural guidance
- Work with enterprise architecture and security teams to align data solutions with data privacy/security policies
- Implement data quality checks, alerting, and lineage tracking using tools such as Great Expectations, Unity Catalog, and data observability platforms
- Build monitoring dashboards, alerts, and recovery mechanisms for critical pipelines
Requirements
- Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience)
- 5+ years of data engineering experience
- 3+ years with Azure Databricks / Apache Spark using PySpark
- Strong experience with Delta Lake, Unity Catalog, Databricks notebooks, and SQL-based warehousing patterns
- Hands‑on experience with ADLS Gen2, Event Hub, ADF, Key Vault, and Synapse
- Strong coding skills in Python and SQL; Scala is a plus
- Understanding of data modeling for OLTP/OLAP, star/snowflake schemas, and performance tuning
- Experience implementing CI/CD for Databricks using Databricks CLI, Terraform, and/or Azure DevOps
- Databricks certification (e.g., Databricks Certified Data Engineer Professional)
- Experience with streaming/real‑time pipelines using Structured Streaming and Event Hub (Kafka experience is referenced)
- ML platform exposure (e.g., MLflow or supporting ML pipelines)
- Reporting/consumption layer experience such as Power BI, Purview, or Microsoft Fabric
- Knowledge of governance/compliance standards including GDPR, HIPAA, and SOC2
- Must currently reside in the Dallas–Fort Worth (DFW) area and be able to work a hybrid schedule in Plano, TX
- Relocation is not available
- Candidates must be authorized to work in the United States without sponsorship now or in the future
Technologies
- Azure Databricks
- PySpark, Spark SQL
- Delta Lake, Unity Catalog
- Python, SQL, Scala
- ADLS Gen2
- Azure Data Factory (ADF)
- Azure Synapse
- Event Hub
- Key Vault, Azure DevOps
- Managed identities, RBAC, masking
- Great Expectations
- Data observability platforms
- Databricks notebooks, Databricks CLI
- Terraform
- Structured Streaming, Kafka
- MLflow
- Power BI, Purview, Microsoft Fabric
Benefits
- Medical, Prescription, Dental & Vision Benefits (for employees working 20+ hours per week)
- Health Savings Account (HSA) (for employees working 20+ hours per week)
- Life & Disability Insurance (for employees working 20+ hours per week)
- MetLife Voluntary Benefits
- Employee Assistance Program (EAP)
- 401K Retirement Savings Plan
- Direct Deposit & weekly epayroll
- Referral Bonus Programs
- Certification and training opportunities
Location: Plano, TX (onsite)
Compensation: USD 105,000 - 150,000 per year