Lead Big Data Engineer: Real-Time Pipelines & Cloud

Compunnel, Inc.; Plano, Northern (TX, KY); Hybrid; USD 140,000 - 180,000; Full time

Job summary

Compunnel, Inc. in Plano, TX seeks a Lead Data Engineer to design and implement scalable data platforms across the enterprise. You will build data pipelines, data models, and governance practices while delivering measurable improvements to developer productivity and data quality.

You will lead cross‑functional teams, work with Snowflake, Databricks, Redshift and cloud platforms, and mentor engineers to raise the bar on performance, reliability, and data-driven decision making.

Qualifications

• 10+ years of application development experience using Python and SQL.
• 5+ years of experience with public cloud platforms such as AWS, GCP, or Azure.
• 5+ years of experience with distributed data and computing technologies such as MapReduce, Hadoop, EMR, Kafka, or Spark.
• 5+ years of experience with data warehouses and services such as Snowflake, Databricks, or Redshift.
• 5+ years of experience with data modeling for data warehousing.
• Strong hands-on experience with Python and SQL.
• Strong understanding of data engineering, data movement, and data pipeline development.
• Experience working with relational and NoSQL databases.
• Experience working in Agile development environments.
• Experience leveraging interactive AI development tools such as Claude Code, Windsurf, or GitHub Copilot.

Responsibilities

• Collaborate with Agile teams to design, develop, test, implement, and support technical solutions using data movement tools and technologies.
• Lead development efforts involving data movement, distributed computing, and full-stack data systems.
• Design and develop scalable data pipelines and data engineering solutions using Python and SQL.
• Work with open-source relational and NoSQL databases and cloud-based data warehousing services.
• Develop and maintain solutions using Snowflake, Databricks, and other modern data platforms.
• Optimize information systems for end users and downstream application consumers using sound data design practices.
• Design and implement data models for data warehousing environments.
• Develop and support real-time data and streaming applications.
• Work with distributed data and computing technologies such as MapReduce, Hadoop, EMR, Kafka, and Spark.
• Apply data quality, governance, and permissible-use principles throughout data engineering activities.
• Leverage interactive AI development tools such as Claude Code, Windsurf, and GitHub Copilot to accelerate software delivery.
• Stay current with emerging technologies and modern data engineering practices.
• Experiment with and evaluate new technologies to improve engineering solutions.
• Participate in internal and external technology communities.
• Mentor other members of the engineering team and contribute to the broader engineering community.

Skills

Python

SQL

Cloud platforms

Distributed computing

Real-time processing

Data modeling

Data warehousing

Agile development

Relational/NoSQL DBs

AI tools

Tools

Snowflake

Databricks

Redshift

Kafka

Spark

Hadoop

MapReduce

AWS

GCP

Azure

GitHub Copilot

Claude Code

Windsurf

Job description

Compunnel, Inc. in Plano, TX seeks a Lead Data Engineer to design and implement scalable data platforms across the enterprise. You will build data pipelines, data models, and governance practices while delivering measurable improvements to developer productivity and data quality.

You will lead cross‑functional teams, work with Snowflake, Databricks, Redshift and cloud platforms, and mentor engineers to raise the bar on performance, reliability, and data-driven decision making.