Senior Data Engineer

Saransh Inc; New York (NY); On-site; USD 120,000 - 150,000; Full time

Job summary

A data engineering firm in New York is seeking a Data Engineer to lead complex data engineering projects. You will design and optimize data architectures in alignment with business needs, and provide technical leadership. The ideal candidate should have over 8 years of IT experience, including 5 years in relevant technologies. Strong proficiency in Python, SQL, and various AWS services is essential, alongside experience in Agile environments. This position involves collaboration with cross-functional teams to ensure the quality and efficiency of data systems.

Qualifications

• 8+ years overall IT experience.
• Minimum 5 years of experience in required technologies.

Responsibilities

• Lead development of large-scale data pipelines and architectures.
• Design, build, and test data architectures aligned with business needs.
• Provide technical leadership and ensure best practices.
• Collaborate with product, finance, and business teams.
• Oversee development of predictive and prescriptive analytics solutions.
• Ensure technical quality and data movement across environments.
• Align data architecture with solution architecture.

Skills

Python scripting

PySpark

SQL

big data processing

data lake processing in Iceberg format

AWS Glue jobs

AWS services (S3, Redshift, Lambda, EMR, Airflow)

BASH/Shell scripting

healthcare finance systems

Agile environments

Job description

Position Summary

Lead complex data engineering projects; design data architectures aligned with business requirements; optimize data workflows and manage data pipelines; ensure smooth operation of data systems.

Responsibilities

• Lead development of large-scale data pipelines and architectures.
• Design, build, and test data architectures aligned with business needs.
• Provide technical leadership and ensure best practices.
• Collaborate with product, finance, and business teams.
• Oversee development of predictive and prescriptive analytics solutions.
• Ensure technical quality and data movement across environments.
• Align data architecture with Healthfirst solution architecture.

Minimum Qualifications

• 8+ years overall IT experience.
• Minimum 5 years of experience in required technologies.

Technical Skills

• Strong experience with Python scripting and PySpark.
• Proficiency in SQL and big data processing.
• Experience with data lake processing in Iceberg format.
• Experience with AWS Glue jobs.
• Experience with AWS services: S3, Redshift, Lambda, EMR, Airflow, Postgres, SNS, EventBridge.
• Expertise in BASH/Shell scripting.
• Strong understanding of healthcare finance systems.
• Experience leading data engineering teams.
• Experience working in Agile environments.