Senior / Lead Big Data Engineer

Compunnel, Inc.; Plano, Northern (TX, KY); Hybrid; USD 140,000 - 180,000; Full time

Job summary

Compunnel, Inc. in Plano, TX seeks a Lead Data Engineer to design and implement scalable data platforms across the enterprise. You will build data pipelines, data models, and governance practices while delivering measurable improvements to developer productivity and data quality.

You will lead cross‑functional teams, work with Snowflake, Databricks, Redshift and cloud platforms, and mentor engineers to raise the bar on performance, reliability, and data-driven decision making.

Qualifications

• 10+ years of application development experience using Python and SQL.
• 5+ years of experience with public cloud platforms such as AWS, GCP, or Azure.
• 5+ years of experience with distributed data and computing technologies such as MapReduce, Hadoop, EMR, Kafka, or Spark.
• 5+ years of experience with data warehouses and services such as Snowflake, Databricks, or Redshift.
• 5+ years of experience with data modeling for data warehousing.
• Strong hands-on experience with Python and SQL.
• Strong understanding of data engineering, data movement, and data pipeline development.
• Experience working with relational and NoSQL databases.
• Experience working in Agile development environments.
• Experience leveraging interactive AI development tools such as Claude Code, Windsurf, or GitHub Copilot.

Responsibilities

• Collaborate with Agile teams to design, develop, test, implement, and support technical solutions using data movement tools and technologies.
• Lead development efforts involving data movement, distributed computing, and full-stack data systems.
• Design and develop scalable data pipelines and data engineering solutions using Python and SQL.
• Work with open-source relational and NoSQL databases and cloud-based data warehousing services.
• Develop and maintain solutions using Snowflake, Databricks, and other modern data platforms.
• Optimize information systems for end users and downstream application consumers using sound data design practices.
• Design and implement data models for data warehousing environments.
• Develop and support real-time data and streaming applications.
• Work with distributed data and computing technologies such as MapReduce, Hadoop, EMR, Kafka, and Spark.
• Apply data quality, governance, and permissible-use principles throughout data engineering activities.
• Leverage interactive AI development tools such as Claude Code, Windsurf, and GitHub Copilot to accelerate software delivery.
• Stay current with emerging technologies and modern data engineering practices.
• Experiment with and evaluate new technologies to improve engineering solutions.
• Participate in internal and external technology communities.
• Mentor other members of the engineering team and contribute to the broader engineering community.

Skills

Python

SQL

Cloud platforms

Distributed computing

Real-time processing

Data modeling

Data warehousing

Agile development

Relational/NoSQL DBs

AI tools

Tools

Snowflake

Databricks

Redshift

Kafka

Spark

Hadoop

MapReduce

AWS

GCP

Azure

GitHub Copilot

Claude Code

Windsurf

Job description

The Lead Data Engineer will be responsible for designing, developing, testing, implementing, and supporting scalable data engineering solutions. The role will focus on building data pipelines, data movement solutions, data modeling, data quality, governance, and developer productivity metrics. As part of the CODE Metrics team, the Lead Data Engineer will help develop a developer productivity metrics platform used by engineers across the enterprise to measure developer effectiveness and experience. The role requires strong expertise in Python, SQL, cloud technologies, distributed computing, real-time data processing, data warehouses, and modern data engineering technologies.

Key Responsibilities

• Collaborate with Agile teams to design, develop, test, implement, and support technical solutions using data movement tools and technologies.
• Lead development efforts involving data movement, distributed computing, and full-stack data systems.
• Design and develop scalable data pipelines and data engineering solutions using Python and SQL.
• Work with open-source relational and NoSQL databases and cloud-based data warehousing services.
• Develop and maintain solutions using Snowflake, Databricks, and other modern data platforms.
• Optimize information systems for end users and downstream application consumers using sound data design practices.
• Design and implement data models for data warehousing environments.
• Develop and support real-time data and streaming applications.
• Work with distributed data and computing technologies such as MapReduce, Hadoop, EMR, Kafka, and Spark.
• Apply data quality, governance, and permissible-use principles throughout data engineering activities.
• Leverage interactive AI development tools such as Claude Code, Windsurf, and GitHub Copilot to accelerate software delivery.
• Stay current with emerging technologies and modern data engineering practices.
• Experiment with and evaluate new technologies to improve engineering solutions.
• Participate in internal and external technology communities.
• Mentor other members of the engineering team and contribute to the broader engineering community.

Required Qualifications

• 10+ years of application development experience using Python and SQL.
• 5+ years of experience working with public cloud platforms such as AWS, GCP, or Azure.
• 5+ years of experience with distributed data and computing technologies such as MapReduce, Hadoop, EMR, Kafka, or Spark.
• 5+ years of experience working with real-time data and streaming applications.
• 5+ years of experience with data warehouses and services such as Snowflake, Databricks, or Redshift.
• 5+ years of experience with data modeling for data warehousing.
• Strong hands-on experience with Python and SQL.
• Strong understanding of data engineering, data movement, and data pipeline development.
• Experience working with relational and NoSQL databases.
• Experience working in Agile development environments.
• Experience leveraging interactive AI development tools such as Claude Code, Windsurf, or GitHub Copilot.
• Strong problem-solving, collaboration, and communication skills.

Preferred Qualifications

Experience with Cube for semantic data layers.

Experience with Tableau or Quick for business intelligence and analytics.