Cloud Big Data Engineer: Scalable Data Pipelines

Delan Associates, Inc; Charlotte (NC); On-site; USD 120,000 - 160,000; Full time

Job summary

Delan Associates, Inc in Charlotte, NC seeks a senior data engineer to design, build, and optimize large-scale data pipelines using GCP, BigQuery, Airflow, and Kafka. You will translate business needs into robust data architectures and collaborate with cross-functional teams.

The role emphasizes hands-on development, scalable data models, CI/CD automation, and a willingness to learn enterprise frameworks for Banking, Financial Crimes and AML platforms.

Qualifications

• 5+ years of hands-on software development experience with Big Data & Analytics solutions.
• Proficiency in SQL and database systems, with experience in designing and optimizing data models for performance and scalability.
• Experience of building event processing pipelines with Kafka or GCP PubSub.
• Design and development experience with Airflow, PubSub Kafka, Git, Jenkins is desirable.
• Knowledge of distributed (multi-tiered) systems, algorithms & relational databases.
• Strong Object-Oriented Programming skills and design patterns.
• Experience with CICD pipelines, Automated test frameworks, and source code management tools (XLR, Jenkins, Git).
• Good knowledge and experience with configuration management tools like GitHub.
• Ability to analyze complex data engineering problems, propose effective solutions, and implement them effectively.

Responsibilities

• Develop and optimize data pipelines for large-scale processing.
• Design data models and optimize performance.
• Collaborate with product teams to deliver Banking and Financial Crimes/AML features.
• Implement CI/CD and automated testing.
• Learn and apply Enterprise Big Data frameworks.

Skills

Big Data & Analytics

SQL

Python

OO Programming

Agile & Scrum

Data pipelines

Communication

Education

BS or MS in CS / CE or related field

Tools

Airflow

Kafka

Pub/Sub

Spark

Hadoop Hive

Git

Jenkins

DataProc

BigQuery

Shell scripting

GitHub

Job description

Delan Associates, Inc in Charlotte, NC seeks a senior data engineer to design, build, and optimize large-scale data pipelines using GCP, BigQuery, Airflow, and Kafka. You will translate business needs into robust data architectures and collaborate with cross-functional teams.

The role emphasizes hands-on development, scalable data models, CI/CD automation, and a willingness to learn enterprise frameworks for Banking, Financial Crimes and AML platforms.