Senior Data Engineer

Apex Systems; The Woodlands (TX); On-site; USD 120,000 - 160,000; Full time

Job summary

A leading company is seeking a Senior Consultant – Databricks Engineer to develop scalable, cloud-based data pipelines for enterprise analytics. The role involves building robust ETL/ELT workflows using Databricks and Spark, with a focus on performance optimization and collaboration within Agile teams.

Qualifications

• 5+ years of experience in data engineering, including 3+ years with Databricks.
• Strong proficiency in PySpark and SQL.
• Experience with cloud environments (Azure or AWS).

Responsibilities

• Design and develop scalable batch and streaming data pipelines in Databricks.
• Build ETL/ELT processes integrating data from diverse sources.
• Optimize data pipelines for performance and cost-efficiency.

Skills

Data Engineering

Apache Spark

PySpark

SQL

ETL/ELT Workflows

Data Architecture

Problem Solving

Communication

Education

Bachelor’s degree in Computer Science

Tools

Git

Databricks

IDMC

Job description

Apex Systems is seeking a Senior Consultant – Databricks Engineer to support the development of scalable, cloud-based data pipelines for enterprise analytics platforms. This role focuses on building robust ETL/ELT workflows using Databricks and Spark, with opportunities to work with advanced features such as Delta Live Tables (DLT) and cloud-native tools like Informatica Intelligent Data Management Cloud (IDMC). The ideal candidate will have hands-on experience building data pipelines from the ground up and strong proficiency in PySpark and SQL.

Key Responsibilities:

1. Design and develop scalable batch and streaming data pipelines in Databricks using PySpark, SQL, and Delta Lake.
2. Build ETL/ELT processes that transform and integrate data from diverse sources into unified analytical datasets.
3. Leverage Delta Live Tables (DLT) and Databricks Workflows to automate and orchestrate data pipelines (preferred).
4. Optimize data pipelines for performance, cost-efficiency, and reusability, applying best practices in Spark tuning and partitioning strategies.
5. Collaborate with architects, data modelers, and analysts to understand data requirements and implement business logic in pipelines.
6. Implement data quality, monitoring, and observability practices across pipelines.
7. Work within Agile delivery teams, participating in sprint planning, code reviews, and deployments.
8. Collaborate with cloud platform teams to ensure proper setup of storage, networking, and security.

Required Skills & Experience:

• 5+ years of experience in data engineering, including 3+ years of hands-on experience with Databricks.
• Strong hands-on experience in Apache Spark and/or Delta Live Tables (DLT) for pipeline development.
• Proficiency in PySpark and SQL for building and optimizing large-scale data pipelines.
• Experience designing and implementing ETL/ELT workflows in cloud environments (Azure or AWS).
• Familiarity with Delta Lake, Unity Catalog, and Databricks Jobs/Workflows.
• Experience working with structured and semi-structured data from APIs, cloud storage, and enterprise systems.
• Solid understanding of modern data architecture concepts (e.g., Medallion architecture, lakehouse patterns).
• Familiarity with Git-based version control, CI/CD practices, and infrastructure automation.
• Excellent problem-solving and communication skills in client-facing environments.
• Bachelor’s degree in Computer Science, Information Systems, or equivalent experience.

Preferred Qualifications:

• Experience with Delta Live Tables (DLT) for declarative pipeline development.
• Exposure to IDMC (Informatica Intelligent Data Management Cloud) or other data integration platforms.
• Databricks or cloud provider certifications (Azure/AWS).
• Familiarity with data governance tools (Unity Catalog, Purview, Collibra) and practices.
• Experience with REST APIs and real-time/streaming architectures (Kafka, Event Hubs, etc.).

EEO Employer

Apex Systems is an equal opportunity employer. We do not discriminate or allow discrimination on the basis of race, color, religion, creed, sex (including pregnancy, childbirth, breastfeeding, or related medical conditions), age, sexual orientation, gender identity, national origin, ancestry, citizenship, genetic information, registered domestic partner status, marital status, disability, status as a crime victim, protected veteran status, political affiliation, union membership, or any other characteristic protected by law. Apex will consider qualified applicants with criminal histories in a manner consistent with the requirements of applicable law. If you require an accommodation in using our website for a search or application, please contact our Employee Services Department at [e-mail] or 844-463-6178.