Cloud BigData Engineer

Delan Associates, Inc; Charlotte (NC); On-site; USD 120,000 - 160,000; Full time

Job summary

Delan Associates, Inc in Charlotte, NC seeks a senior data engineer to design, build, and optimize large-scale data pipelines using GCP, BigQuery, Airflow, and Kafka. You will translate business needs into robust data architectures and collaborate with cross-functional teams.

The role emphasizes hands-on development, scalable data models, CI/CD automation, and a willingness to learn enterprise frameworks for Banking, Financial Crimes and AML platforms.

Qualifications

• 5+ years of hands-on software development experience with Big Data & Analytics solutions.
• Proficiency in SQL and database systems, with experience in designing and optimizing data models for performance and scalability.
• Experience of building event processing pipelines with Kafka or GCP PubSub.
• Design and development experience with Airflow, PubSub Kafka, Git, Jenkins is desirable.
• Knowledge of distributed (multi-tiered) systems, algorithms & relational databases.
• Strong Object-Oriented Programming skills and design patterns.
• Experience with CICD pipelines, Automated test frameworks, and source code management tools (XLR, Jenkins, Git).
• Good knowledge and experience with configuration management tools like GitHub.
• Ability to analyze complex data engineering problems, propose effective solutions, and implement them effectively.

Responsibilities

• Develop and optimize data pipelines for large-scale processing.
• Design data models and optimize performance.
• Collaborate with product teams to deliver Banking and Financial Crimes/AML features.
• Implement CI/CD and automated testing.
• Learn and apply Enterprise Big Data frameworks.

Skills

Big Data & Analytics

SQL

Python

OO Programming

Agile & Scrum

Data pipelines

Communication

Education

BS or MS in CS / CE or related field

Tools

Airflow

Kafka

Pub/Sub

Spark

Hadoop Hive

Git

Jenkins

DataProc

BigQuery

Shell scripting

GitHub

Job description

Job Description:

Interviews: Live, Technical Hands On

BS or MS degree in computer science, computer engineering, or other technical discipline, or equivalent work experience.

5+ years of hands-on software development experience with Big Data & Analytics solutions

GCP Cloud - Big Query, Airflow, DataProc, PubSub, Hadoop Hive, Spark, Python, shell scripting,

Strong experience in designing, developing, and optimizing data pipelines for large-scale data processing, transformation, and analysis using Big Data and GCP technologies.

Proficiency in SQL and database systems, with experience in designing and optimizing data models for performance and scalability.

Experience of building event processing pipelines with Kafka or GCP PubSub.

Design and development experience with Airflow, PubSub Kafka, Git, Jenkins is desirable.

Knowledge of distributed (multi-tiered) systems, algorithms & relational databases.

Strong Object-Oriented Programming skills and design patterns.

Experience with CICD pipelines, Automated test frameworks, and source code management tools (XLR, Jenkins, Git).

Good knowledge and experience with configuration management tools like GitHub

Ability to analyze complex data engineering problems, propose effective solutions, and implement them effectively.

Ability and willingness to learn, adopt and build solutions using the Enterprise frameworks established for Big Data

Looks proactively beyond the obvious for continuous improvement opportunities.

Communicates effectively with product and cross functional team.

Willingness to learn new technologies and leverage them to their optimal potential.

Understanding of various SDLC methodologies, familiarity with Agile & scrum ceremonies.

Certifications in cloud platform (GCP Professional Data Engineer) is a plus.

Provide engineering expertise to the team for developing features for Banking including Financial Crimes, Anti-money Laundering platforms

Improve engineering practices, including driving resilience, predictability, and end efficiency through automation

Partner with product teams to envision and deliver solutions which transform Financial Crimes Compliance using technology