Principal Software Engineer Data Platform Engineering

Compunnel, Inc.; Raleigh (NC); On-site; USD 120,000 - 150,000; Full time

Job summary

A technology solutions provider is seeking a Principal Software Engineer - Data Platform Engineering in Raleigh, North Carolina. This role involves designing scalable data pipelines and developing micro-services using Python, Apache Spark, and Databricks. The ideal candidate will have at least 10 years of software engineering experience, with a strong focus on data platforms. This opportunity offers an engaging work environment with a focus on cloud-native technologies.

Qualifications

• 10+ years of experience in software engineering, with at least 3 years in data engineering.
• Proficiency in Python with strong knowledge of Spark technologies.
• Experience writing infrastructure-as-code using Terraform.

Responsibilities

• Design and implement scalable data pipelines for IoT telemetry data.
• Analyze and optimize Spark jobs for performance.
• Develop Python-based micro-services and support data workflows.

Skills

Python

PySpark

Apache Spark

Databricks

Terraform

Docker

CI/CD Pipelines

Agile/Scrum

Git

Education

Bachelor’s degree in Computer Science or related field

Job description

Principal Software Engineer Data Platform Engineering

08/19/2025

Full Time

Active

Job Description:

We are enabling the transformation of how energy and water are managed. As part of this mission, we are investing in cloud-native data platform technologies that power intelligent analytics at scale — particularly for large IoT data workloads. We are looking for a Principal Software Engineer - Data Platform Engineering with a strong background in data engineering to help design, build, and scale our next-generation data platform and services.

This role will focus on developing performant, scalable, and secure data analytics and data pipelines using Python, Apache Spark, Databricks, and Azure-native technologies. The ideal candidate is passionate about working with large-scale datasets, understands Spark query execution plans, and has experience deploying infrastructure using Terraform and container technologies like Docker.

Duties & Responsibilities:

• Design and implement scalable data pipelines for high-volume IoT telemetry data using PySpark, Spark SQL, and Databricks
• Analyze and optimize Spark jobs using execution plans, caching strategies, and memory tuning
• Develop Python-based micro-services and reusable libraries to support data processing workflows
• Work in a DevOps-driven Agile/Scrum environment to deliver high-quality code in fast iterations
• Collaborate with product managers, data scientists, and software engineers to define system requirements and data integration needs
• Build and manage infrastructure as code using Terraform to deploy workloads on Azure
• Create CI/CD pipelines for deploying and testing data workflows and services
• Package and deploy applications in Docker containers and manage runtime environments
• Apply strong testing practices, including unit tests, integration tests, and test automation
• Participate in architectural discussions and technical design reviews
• Ensure secure, reliable, and cost-effective operation of data workloads on cloud platforms

Required Skills & Experience:

• Minimum of 10 years of software engineering experience, with at least 3 years focused on data platform or data engineering roles
• Proficient in Python, with strong knowledge of PySpark and Apache Spark
• Hands-on experience with Databricks and Spark SQL
• Strong understanding of Spark performance tuning and query execution plans
• Experience writing infrastructure-as-code using Terraform
• Working knowledge of Docker and containerized deployment
• Experience with CI/CD pipelines, version control (e.g., Git), and automated testing
• Experience with Agile/Scrum methodologies
• Bachelor’s degree in Computer Science, Engineering, or related field — or equivalent experience

Preferred Skills & Experience:

• Experience with large-scale IoT or telemetry data processing
• Familiarity with Kafka, Event Hubs, or other streaming platforms
• Familiarity with cost optimization in cloud-based data processing
• Experience with monitoring and observability tools like Prometheus, Grafana, or Datadog

Email ID * This field is required Please enter valid emailId.

Cell phone * This field is required Please enter valid cell phone.

First Name * This field is required Please enter valid first name.

Last Name * This field is required Please enter valid last name.