Lead Data Engineer

The Phoenix Group; New York (NY); On-site; USD 140,000 - 190,000; Full time

Job summary

The Phoenix Group is seeking a Lead Data Engineer to own design, development, and maintenance of enterprise-scale data pipelines and architecture. You will collaborate with cross-functional teams to enable analytics, AI/ML initiatives, and governance across cloud environments.

Responsibilities include building scalable pipelines with Snowflake, Airflow, and dbt, implementing medallion data models, and driving CI/CD for data solutions while ensuring security and observability.

Qualifications

• 5+ years in data or software engineering with progressive responsibilities.
• Experience with Snowflake, AWS, and data orchestration tools.
• Designing data architectures and medallion models.
• Migration of production workflows to cloud orchestration.
• Experience with IAM, S3, Lambda, and CloudWatch.
• Data quality, lineage, and observability practices.
• Strong communication with technical and business stakeholders.

Responsibilities

• Design, build, test, deploy, and support scalable data pipelines using Airflow, dbt, Snowflake.
• Optimize cloud data architecture, schemas, security roles, and performance.
• Implement medallion architecture (raw/bronze to gold) for data.
• Migrate legacy workflows to cloud orchestration with reusable patterns.
• Maintain orchestration runtimes: schedulers, workers, metadata, queues, autoscaling.
• Collaborate with AWS services like S3, IAM, EKS, Lambda, CloudWatch.
• Build secure integrations across APIs, SFTP, databases, files, and event systems.
• Develop ingestion pipelines for structured and unstructured data incl OCR.
• Embed data quality, lineage, observability into workflows.
• Partner with governance and security teams on data ownership and retention.
• Build CI/CD for workflows and infrastructure code with testing and rollback.
• Provide on-call support and incident management.
• Establish engineering standards and reusable frameworks.
• Lead technical workstreams and code reviews.

Skills

Snowflake

Airflow

dbt

Python

SQL

Kubernetes

CI/CD

Data governance

Tools

Git workflows

Docker

Kubernetes

Job description

The Lead Data Engineer will own the design, development, and maintenance of robust data architecture and pipelines leveraging cloud-based technologies, supporting strategic analytics, AI/ML initiatives, and data governance initiatives across the organization's enterprise-scale environment. Collaborating with cross-functional teams, this individual will ensure high-quality, secure, and performant data solutions that enable decision-making and operational excellence.

Key Responsibilities

• Design, build, test, deploy, and support scalable data pipelines utilizing orchestration tools (Airflow, Astronomer), data transformation frameworks (dbt), cloud data platforms (Snowflake), SQL, and Python.
• Develop and optimize cloud-based data architecture, including database schemas, data warehouses, security roles, workload isolation, cost management, and performance tuning within Snowflake.
• Implement medallion data architecture principles - raw/bronze, silver, and gold layers - for structured and semi-structured data management.
• Lead migration and modernization of legacy workflows to cloud orchestration platforms, establishing reusable patterns for dependency management, retries, backfills, alerting, recovery, and ensuring idempotency.
• Troubleshoot and maintain the full runtime environment of orchestration platforms, including schedulers, workers, executors, metadata stores, queues, and autoscaling components.
• Collaborate with cloud infrastructure teams to manage AWS services such as S3, IAM, EKS, ECS, Fargate, EC2, Lambda, Secrets Manager, KMS, CloudWatch, and networking components.
• Build secure, resilient integrations across APIs, SFTP, databases, cloud storage, vendor APIs, flat files, Excel files, and event-driven systems.
• Develop ingestion and processing pipelines for structured, semi-structured, and unstructured data, including OCR/image-based document extraction.
• Embed data quality testing, monitoring, lineage, reconciliation, and observability into data workflows to ensure reliability and compliance.
• Partner with Data Governance, Security, and Business teams to enforce data ownership, classification, access policies, retention, and auditing standards.
• Build and maintain CI/CD pipelines for orchestration workflows, transformations, and infrastructure code, ensuring automated testing, environment promotion, deployment, and rollback capabilities.
• Support production environments through on-call and incident management processes, conducting root cause analysis, stakeholder communication, and corrective actions.
• Establish reusable frameworks, engineering standards, and best practices to improve platform reliability, consistency, and delivery velocity.
• Lead technical workstreams, mentoring engineering team members through design reviews, code reviews, documentation, and troubleshooting.

Core Qualifications & Requirements

• Minimum of 5+ years in data engineering or software engineering roles with progressive responsibilities.
• Proven experience with cloud-based data platforms, including Snowflake, AWS cloud infrastructure, and data orchestration tools such as Airflow/Astronomer.
• Strong expertise in designing and implementing data architectures, including dimensional and medallion models.
• Hands-on experience migrating and maintaining production workflows in orchestration platforms.
• Deep understanding of cloud compute environments, including Kubernetes, containers, schedulers, workers, and autoscaling architectures.
• Proficiency in AWS services such as S3, IAM, EKS/ECS, Lambda, Secrets Manager, KMS, and CloudWatch for security, storage, and monitoring.
• Skilled in integrating complex data sources and systems, including APIs, SFTP, relational and NoSQL databases, cloud storage, and file-based workflows.
• Experience processing semi-structured and unstructured data, including OCR, document extraction, and image processing.
• Strong background in implementing data quality controls, automated testing, data lineage, and observability tools.
• Knowledge of data governance principles, access controls, data classification, retention policies, and audit requirements.
• Experience with CI/CD pipelines, Git workflows, and automated deployment practices for data solutions and infrastructure.
• Excellent troubleshooting skills across cloud infrastructure, data pipelines, application code, and security permissions.
• Ability to lead technical initiatives and communicate effectively with both technical teams and business stakeholders.