Data Engineer

ATC; New York (NY); On-site; USD 120,000 - 180,000; Full time

Job summary

ATC in New York seeks an experienced Data Engineer to design, build, and optimize scalable data pipelines and cloud-based platforms. You will implement ETL/ELT workflows with Databricks and Apache Spark and collaborate with analysts to deliver reliable data solutions.

The role requires 5+ years, strong Python and SQL skills, hands-on cloud experience (AWS/Azure/Google), and a focus on data quality, governance, monitoring, and CI/CD practices.

Qualifications

• Bachelor's or Master’s degree in Computer Science, Information Technology, Engineering, or a related field.
• 5+ years of experience as a Data Engineer.
• Strong programming skills in Python.
• Advanced SQL proficiency.
• Hands-on experience with Databricks.
• Strong experience with Apache Spark (PySpark preferred).
• Experience building and maintaining ETL/ELT pipelines.
• Experience with Apache Airflow or similar workflow orchestration tools.
• Experience with data warehousing technologies such as Snowflake, Amazon Redshift, Google BigQuery, or Azure Synapse Analytics.
• Hands-on experience with at least one cloud platform (AWS, Azure, or Google Cloud Platform).
• Experience with Git, Docker, and CI/CD pipelines.
• Knowledge of relational and NoSQL databases.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Build and optimize data ingestion, transformation, and integration workflows using Databricks and Apache Spark.
• Develop and maintain data lakes and cloud-based data warehouses.
• Create scalable batch and real-time data processing solutions.
• Optimize data pipelines for performance, reliability, and scalability.
• Develop reusable data engineering frameworks and automation solutions.
• Collaborate with data analysts, data scientists, and business stakeholders to deliver high-quality data solutions.
• Implement data quality, governance, monitoring, and security best practices.
• Troubleshoot production issues and continuously improve data platform performance.
• Follow DevOps and CI/CD best practices for data engineering projects.

Skills

Python

SQL

Databricks

Apache Spark

ETL/ELT development

Git

Docker

CI/CD

Airflow

NoSQL databases

Education

Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field

Tools

Databricks

Apache Spark

Airflow

Snowflake/Redshift/BigQuery/Azure Synapse

Git

Docker

Terraform

Job description

We are seeking a Data Engineer with 5+ years of experience in designing, developing, and optimizing scalable data pipelines and cloud-based data platforms. The ideal candidate will have strong expertise in Python, SQL, Databricks, Apache Spark, and ETL/ELT development, along with hands-on experience in at least one cloud platform (AWS, Azure, or Google Cloud Platform).

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Build and optimize data ingestion, transformation, and integration workflows using Databricks and Apache Spark.
• Develop and maintain data lakes and cloud-based data warehouses.
• Create scalable batch and real-time data processing solutions.
• Optimize data pipelines for performance, reliability, and scalability.
• Develop reusable data engineering frameworks and automation solutions.
• Collaborate with data analysts, data scientists, and business stakeholders to deliver high-quality data solutions.
• Implement data quality, governance, monitoring, and security best practices.
• Troubleshoot production issues and continuously improve data platform performance.
• Follow DevOps and CI/CD best practices for data engineering projects.

Required Qualifications

• Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field.
• 5+ years of experience as a Data Engineer.
• Strong programming skills in Python.
• Advanced SQL proficiency.
• Hands‑on experience with Databricks.
• Strong experience with Apache Spark (PySpark preferred).
• Experience building and maintaining ETL/ELT pipelines.
• Experience with Apache Airflow or similar workflow orchestration tools.
• Experience with data warehousing technologies such as Snowflake, Amazon Redshift, Google BigQuery, or Azure Synapse Analytics.
• Hands‑on experience with at least one cloud platform (AWS, Azure, or Google Cloud Platform).
• Experience with Git, Docker, and CI/CD pipelines.
• Knowledge of relational and NoSQL databases.

Preferred Qualifications

• Experience with Kafka or other streaming platforms.
• Experience with Delta Lake.
• Familiarity with Apache Iceberg or Apache Hudi.
• Experience with Infrastructure as Code (Terraform or CloudFormation).
• Knowledge of data governance and data quality frameworks.