Senior Data Engineer

Compunnel, Inc.; Charlotte (NC); On-site; USD 120,000 - 150,000; Full time

Job summary

Compunnel, Inc. is seeking a Senior Data Engineer to design, develop, and optimize scalable cloud-based data platforms essential for analytics and business intelligence. The ideal candidate will lead the development of high-performance ETL/ELT pipelines and ensure compliance with security and regulatory standards.

The position requires expertise in Python, SQL, and cloud platforms like AWS or Azure, along with a strong foundation in data engineering methodologies.

Qualifications

• 6+ years of experience in Data Engineering or ETL development.
• Strong SQL skills with a proven track record of optimizing complex queries.
• Hands-on experience with Python for data processing and automation.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines.
• Collaborate with cross-functional teams for high-quality data solutions.
• Monitor production data pipelines and troubleshoot failures.

Skills

Python

SQL

Cloud data platforms

ETL/ELT development

Data engineering best practices

Apache Spark

Data quality validation

Agile methodologies

Education

Bachelor's degree in Computer Science or Information Systems

Tools

AWS

Azure

GCP

Informatica

Talend

Apache Airflow

Kafka

Job description

The Senior Data Engineer will design, develop, and optimize scalable cloud-based data platforms that support enterprise analytics, regulatory reporting, business intelligence, and machine learning initiatives. This role is responsible for building high-performance ETL/ELT pipelines, implementing data engineering best practices, and delivering reliable, high-quality datasets across cloud environments. The ideal candidate will have strong expertise in Python, SQL, cloud data platforms, distributed data processing, and modern data engineering technologies.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines to ingest data from multiple internal and external sources.
• Build and optimize cloud-based data pipelines using modern data engineering frameworks and orchestration tools.
• Develop data transformation workflows for structured, semi-structured, and unstructured data.
• Design, develop, and maintain data models supporting analytics, reporting, and machine learning initiatives.
• Collaborate with data architects, business analysts, data scientists, and cross-functional teams to deliver high-quality data solutions.
• Implement data quality validation, metadata management, and data governance standards.
• Optimize SQL queries and distributed processing jobs for performance and scalability.
• Develop reusable data engineering frameworks, automation scripts, and best practices.
• Support batch and near real-time data processing requirements.
• Monitor production data pipelines, troubleshoot failures, and participate in production support and on-call rotations.
• Participate in Agile ceremonies, architecture discussions, code reviews, and technical design sessions.
• Ensure compliance with enterprise security, privacy, and regulatory standards.
• Create and maintain technical documentation for data pipelines, processes, and architecture.

Required Qualifications

• Bachelor's degree in Computer Science, Information Systems, or a related field.
• 6+ years of experience in Data Engineering or ETL development.
• Strong SQL skills with experience designing, optimizing, and troubleshooting complex queries.
• Hands-on experience with Python for data processing, automation, and pipeline development.
• Experience with cloud platforms such as AWS, Azure, or GCP.
• Experience with ETL tools such as Informatica, Talend, DataStage, AWS Glue, Azure Data Factory, or similar technologies.
• Experience working with analytical databases such as Snowflake, Amazon Redshift, Azure Synapse, BigQuery, or equivalent.
• Experience with Apache Spark or PySpark for distributed data processing.
• Experience using Git and CI/CD practices for source control and deployment automation.
• Strong understanding of data warehousing concepts, dimensional modeling, and data architecture.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and collaboration skills.

Preferred Qualifications

• Experience with Apache Airflow, Control-M, or other workflow orchestration platforms.
• Experience with Kafka or other streaming technologies.
• Experience with Databricks and Delta Lake.
• Experience working in financial services, banking, insurance, risk management, or related industries.
• Familiarity with data governance, data lineage, and metadata management tools.
• Experience supporting machine learning or AI data pipelines.
• Experience working in Agile/Scrum development environments.