Lead Data Engineer

jobbot; Grand Prairie (TX); Remote; USD 150,000 - 190,000; Full time

Job summary

An established industry player is on the lookout for a Lead Data Engineer to join their innovative team. This full-time remote role offers a competitive salary and the opportunity to make a significant impact in the financial services sector. You'll be at the forefront of designing and implementing robust data pipelines, collaborating with cross-functional teams to drive data-driven solutions. If you're passionate about leveraging cutting-edge technologies and enhancing data engineering practices, this role is tailored for you. Join a dynamic environment where your expertise will be valued and your contributions will shape the future of data engineering.

Qualifications

• 5+ years in data engineering with proven data pipeline development.
• Strong background in relational databases and cloud-native solutions.

Responsibilities

• Lead design and implementation of end-to-end data pipelines.
• Collaborate with teams to develop and deploy data solutions.

Skills

Data Pipeline Design

ETL Processes

Python

SQL

Cloud Platforms (Azure, AWS)

Data Architecture

Dimensional Modeling (Kimball)

API Development

Problem-Solving

Tools

Apache Airflow

Docker

Kubernetes

Snowflake

Fabric

DBT

PySpark

Job description

Our client, in the financial services industry, is seeking a Lead Data Engineer to join their team. This is a full-time, direct hire, remote role that can pay $150-190K base salary plus benefits, depending on experience.

This role is ideal for someone who thrives in a dynamic, fast-paced environment, enjoys solving complex data problems, and is passionate about driving innovation in data engineering. If you're looking to make an impact on the financial landscape with cutting-edge data solutions, this could be for you!

Core Responsibilities:

1. Lead the design and implementation of end-to-end data pipelines, from extraction (API, scraping, pyodbc) to cleansing/transformation (Python, TSQL) and loading into SQL databases or data lakes.
2. Oversee the development of robust data architectures that support efficient querying and analytics, ensuring high-performance and scalable data workflows.
3. Collaborate with data scientists, software developers, business intelligence teams, and stakeholders to develop and deploy data solutions that meet business needs.
4. Ensure smooth coordination between engineering and other teams to translate business requirements into technical solutions.
5. Guide the development of data models and business schemas, ensuring they are optimized for both relational (3NF) and dimensional (Kimball) architectures.
6. Lead the creation of scalable, reliable data models and optimize them for performance and usability.
7. Develop and maintain infrastructure for large-scale data solutions, leveraging cloud platforms (Azure, AWS) and containerization technologies (Docker, Kubernetes).
8. Lead the use of modern data platforms such as Snowflake and Fabric, ensuring their effective utilization in large-scale data solutions.
9. Manage and optimize data pipelines using tools like Apache Airflow, Prefect, DBT, and SSIS, ensuring efficiency, scalability, and reliability of ETL processes.
10. Ensure robust testing, monitoring, and validation of all data systems and pipelines.
11. Drive continuous improvement in data engineering processes and practices, aligning with industry best practices.
12. Foster a culture of clean code, best practices, and rigorous testing across the team.
13. Demonstrate strong experience with data pipeline design and implementation, including ETL processes.
14. Proficiency in SQL (Postgres, SQL Server) and experience with modern data warehouse solutions (Snowflake, Fabric).
15. Expertise in Python for data engineering tasks, including data manipulation (Pandas, NumPy) and workflow management (Dask, PySpark, FastAPI).
16. Solid knowledge of cloud platforms (Azure, AWS) and big data technologies (Hadoop, Spark).
17. Hands-on experience with Docker, Kubernetes, and containerized environments.
18. Strong understanding of dimensional modeling (Kimball), relational database design (3NF), and data architecture best practices.
19. Experience with API development and management.
20. Proficiency with orchestration tools like Prefect or Airflow.
21. Focus on testing and validation to ensure reliability and performance.

Experience & Qualifications:

• 5+ years in data engineering roles, with a proven track record in developing and maintaining data pipelines and architectures.
• Experience working with large-scale data platforms and cloud environments.
• Strong background in relational databases, dimensional data modeling, and cloud-native solutions.
• Familiarity with data engineering tools such as Apache Airflow, Prefect, and cloud storage platforms.
• Excellent problem-solving skills for navigating complex technical challenges.