Senior Data Engineer — Spark, Databricks & Cloud Pipelines

ATC; New York (NY); On-site; USD 140,000 - 180,000; Full time

Job summary

ATC is seeking a Data Engineer with 5+ years of experience to design, develop, and optimize scalable data pipelines and cloud-based platforms. The role focuses on Python, SQL, Databricks, Apache Spark, and ETL/ELT development, with hands-on work on AWS, Azure, or Google Cloud Platform.

You will build data lakes and cloud warehouses, implement data quality and security practices, collaborate with analysts and scientists, and follow DevOps and CI/CD for data projects in a fast-growing environment.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field.
• 5+ years of experience as a Data Engineer.
• Strong programming skills in Python.
• Advanced SQL proficiency.
• Hands-on experience with Databricks.
• Strong experience with Apache Spark (PySpark preferred).
• Experience building and maintaining ETL/ELT pipelines.
• Experience with Apache Airflow or similar workflow orchestration tools.
• Experience with data warehousing technologies such as Snowflake, Amazon Redshift, Google BigQuery, or Azure Synapse Analytics.
• Hands-on experience with at least one cloud platform (AWS, Azure, or Google Cloud Platform).
• Experience with Git, Docker, and CI/CD pipelines.
• Knowledge of relational and NoSQL databases.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Build and optimize data ingestion, transformation, and integration workflows using Databricks and Apache Spark.
• Develop and maintain data lakes and cloud-based data warehouses.
• Create scalable batch and real-time data processing solutions.
• Optimize data pipelines for performance, reliability, and scalability.
• Develop reusable data engineering frameworks and automation solutions.
• Collaborate with data analysts, data scientists, and business stakeholders to deliver high-quality data solutions.
• Implement data quality, governance, monitoring, and security best practices.
• Troubleshoot production issues and continuously improve data platform performance.
• Follow DevOps and CI/CD best practices for data engineering projects.

Skills

Python programming

SQL proficiency

ETL/ELT development

Cloud platform experience

Education

Bachelor’s or Master’s in CS/IT/Engineering

Tools

Databricks

Apache Spark

Git

Docker

CI/CD pipelines

Apache Airflow

Snowflake

Redshift

BigQuery

Azure Synapse Analytics

Terraform

CloudFormation

Job description

ATC is seeking a Data Engineer with 5+ years of experience to design, develop, and optimize scalable data pipelines and cloud-based platforms. The role focuses on Python, SQL, Databricks, Apache Spark, and ETL/ELT development, with hands-on work on AWS, Azure, or Google Cloud Platform.

You will build data lakes and cloud warehouses, implement data quality and security practices, collaborate with analysts and scientists, and follow DevOps and CI/CD for data projects in a fast-growing environment.