ADF Data Engineer

Veriipro; Houston (TX); On-site; USD 110,000 - 150,000; Full time

Job summary

Veriipro in Houston, TX seeks a Senior Data Engineer with Azure expertise to design and optimize data pipelines using Azure Data Factory and Databricks. You will code in Python/Scala, tune SQL, validate data, perform UI testing, and collaborate with stakeholders to deliver scalable data solutions for large datasets.

Requires 8–10 years overall experience, 3+ years Azure, and on-site work. This role offers growth opportunities and exposure to modern cloud data platforms at Veriipro.

Qualifications

• 8-10 years of overall experience.
• 3+ years of Azure experience.
• Strong skills in Azure Data Factory pipelines and Databricks.
• Expert SQL with large datasets.

Responsibilities

• Coding & Development: Design, develop, and optimize ADF/Databricks data pipelines and solutions.
• UI Testing: Perform functional/UI testing, data validation, and troubleshooting.
• Stakeholder Management: Collaborate with stakeholders, understand requirements, and communicate project status/issues effectively.

Skills

Azure Data Factory

Azure Databricks

Python/Scala

SQL

Delta Lake

Git

CI/CD

IaC (IaC exposure)

Job description

Experience: 8-10 years overall, with 3+ years of Azure experience.

Required Skills

• Strong hands-on experience with Azure Data Factory (ADF) - pipelines, data flows, triggers, integration runtimes, parameterization, error handling, and retries.
• Strong experience with Azure Databricks - Notebooks, Jobs/Workflows, PySpark/Scala, Delta Lake, cluster configuration, and optimization.
• Advanced Python or Scala with strong coding standards, modularization, and unit testing (pytest).
• Expert SQL - CTEs, window functions, query tuning, and large datasets.
• Experience with ADLS Gen2, Delta Lake, Parquet, partitioning, Z-Ordering, OPTIMIZE, and VACUUM.
• Experience with Git, branching strategies, and CI/CD for ADF/Databricks.
• Exposure to Infrastructure as Code (IaC) is a plus.

Roles and Responsibilities

• Coding & Development: Design, develop, and optimize ADF/Databricks data pipelines and solutions.
• UI Testing: Perform functional/UI testing, data validation, and troubleshooting.
• Stakeholder Management: Collaborate with stakeholders, understand requirements, and communicate project status/issues effectively.