Data Engineering Specialist

Compunnel, Inc.; Dallas (TX); On-site; USD 110,000 - 140,000; Full time

Job summary

A leading tech firm is seeking a Data Engineering Specialist in Dallas, TX, to design scalable Lakehouse solutions and optimize data processing using Azure technologies. The ideal candidate will possess expertise in Azure Databricks and Python, and have strong collaboration skills. Responsibilities include guiding junior members and participating in Agile processes. Competitive salary and growth opportunities available.

Qualifications

• 5+ years of experience in Azure Databricks with PySpark.
• 4+ years of experience in Azure Data Factory (ADF).
• 3+ years of experience in Python programming.
• Strong understanding of Spark optimization and performance tuning.

Responsibilities

• Design, develop, and maintain Lakehouse solutions using Azure Databricks.
• Optimize Spark jobs and manage large-scale data processing.
• Govern and manage data access using Unity Catalog.
• Build modular and reusable workflows using Azure Data Factory.
• Implement secure data lake storage using ADLS Gen2.

Skills

Azure Databricks

PySpark

Azure Data Factory

Python programming

CI/CD tools

Git

Tools

Azure DevOps

Terraform

SonarQube

Job description

Overview

We are seeking a highly skilled Data Engineering Specialist with expertise in Azure Cloud and DevOps practices.

The ideal candidate will be passionate about building scalable data solutions, optimizing performance, and collaborating with cross-functional teams to deliver high-quality products.

This role involves working on Lakehouse architectures, data orchestration, and secure data management using modern tools and frameworks.

Key Responsibilities

• Design, develop, and maintain Lakehouse solutions using Azure Databricks and PySpark.
• Optimize Spark jobs and manage large-scale data processing using RDD/DataFrame APIs.
• Govern and manage data access using Unity Catalog, including permissions, lineage, and audit trails.
• Build modular and reusable workflows using Azure Data Factory and Databricks Workflows.
• Implement secure, hierarchical namespace-based data lake storage using ADLS Gen2.
• Develop T-SQL queries, stored procedures, and manage metadata layers on Azure SQL.
• Work across the Azure ecosystem including networking, security, monitoring, and cost management.
• Write modular, testable Python code for data transformations and reusable components.
• Lead solution design discussions, prepare technical documentation, and mentor junior team members.
• Ensure adherence to coding guidelines, design patterns, and peer review processes.
• Collaborate with stakeholders and cross-functional teams to translate requirements into deliverables.
• Participate in Agile/Scrum processes and provide regular updates on progress and issues.

Required Qualifications

• 5+ years of experience in Azure Databricks with PySpark, Databricks Workflows, Unity Catalog, and Azure Cloud.
• 4+ years of experience in Azure Data Factory (ADF), ADLS Gen2, and Azure SQL.
• 3+ years of experience in Python programming and package development.
• Strong understanding of Spark optimization, file formats (Parquet/Delta), and performance tuning.
• Experience with CI/CD tools (Azure DevOps, Terraform, ARM/Bicep).
• Familiarity with version control systems (Git), code quality tools (SonarQube, pylint), and testing frameworks (Pytest).
• Excellent communication and collaboration skills.
• Experience preparing HLD/LLD and architecture diagrams.
• Exposure to Agile tools like Jira or Azure DevOps.

Preferred Qualifications

• Experience with Azure Entra/AD and GitHub Actions.
• Orchestration experience using Airflow, Dagster, or Logic Apps.
• Exposure to event-driven architectures using Kafka, Azure Event Hub, or Google Cloud Pub/Sub.
• Experience with Change Data Capture (CDC) solutions using Debezium.
• Hands-on experience with Azure Synapse and Databricks Lakehouse migration projects.
• Experience managing cloud storage solutions on Azure and Google Cloud.