Senior Data Engineer: Lead Big Data Pipelines & Architecture

Saransh Inc; New York (NY); On-site; USD 120,000 - 150,000; Full time

Job summary

A data engineering firm in New York is seeking a Data Engineer to lead complex data engineering projects. You will design and optimize data architectures in alignment with business needs, and provide technical leadership. The ideal candidate should have over 8 years of IT experience, including 5 years in relevant technologies. Strong proficiency in Python, SQL, and various AWS services is essential, alongside experience in Agile environments. This position involves collaboration with cross-functional teams to ensure the quality and efficiency of data systems.

Qualifications

• 8+ years overall IT experience.
• Minimum 5 years of experience in required technologies.

Responsibilities

• Lead development of large-scale data pipelines and architectures.
• Design, build, and test data architectures aligned with business needs.
• Provide technical leadership and ensure best practices.
• Collaborate with product, finance, and business teams.
• Oversee development of predictive and prescriptive analytics solutions.
• Ensure technical quality and data movement across environments.
• Align data architecture with solution architecture.

Skills

Python scripting

PySpark

SQL

big data processing

data lake processing in Iceberg format

AWS Glue jobs

AWS services (S3, Redshift, Lambda, EMR, Airflow)

BASH/Shell scripting

healthcare finance systems

Agile environments

Job description

A data engineering firm in New York is seeking a Data Engineer to lead complex data engineering projects. You will design and optimize data architectures in alignment with business needs, and provide technical leadership. The ideal candidate should have over 8 years of IT experience, including 5 years in relevant technologies. Strong proficiency in Python, SQL, and various AWS services is essential, alongside experience in Agile environments. This position involves collaboration with cross-functional teams to ensure the quality and efficiency of data systems.