Senior Data Engineer

Aptonet; Atlanta (GA); On-site; USD 110,000 - 150,000; Full time

Benefits offered by this job

Health insurance

Health savings account

Vision insurance

Flexible spending accounts

Life insurance

Job summary

Aptonet is seeking a Data Engineer in Atlanta, GA to build end-to-end data pipelines, manage ingestion, transformation, and loading, and enable analytics across large data warehouses. The role emphasizes cloud migration, Databricks, and robust data governance.

You will design robust ETL processes, ensure data quality, and collaborate with cross-functional teams to support data-driven decision making. This is a full-time, on-site position with competitive benefits.

Qualifications

• Bachelor's or Master's degree in Computer Science or related field.
• Proven experience building end-to-end data pipelines.
• Cloud migration experience (AWS, Azure or Google Cloud).
• Experience with Databricks and Apache Spark for large-scale data processing.
• In-depth knowledge of ETL processes for large-scale data warehouses.
• Knowledge of databases such as Oracle DB, DB2 and Teradata (AWS preferred).

Responsibilities

• Design, implement, and maintain end-to-end data pipelines covering ingestion to loading.
• Develop and manage data ingestion from multiple sources, ensuring reliability.
• Perform data transformations and enrichment for usable analytics-ready data.
• Implement data loading into data warehouses or data lakes with quality controls.
• Design data validation mechanisms to ensure accuracy and completeness.
• Utilize Spark-based processing to handle large data volumes efficiently.
• Monitor data pipelines and proactively address issues; optimize performance.
• Plan and execute cloud migration strategies with security and cost controls.
• Leverage Databricks for scalable data processing and analytics.
• Apply ETL best practices and govern data movement across warehouses.
• Collaborate with cross-functional teams to support data-driven decisions.

Skills

Databricks

AWS migration

End-to-end pipelines

ETL processes

Data governance

Spark

Python

SQL

Cloud migration

Teradata

Education

Bachelor's or Master's degree in CS/IS

Tools

Databricks

Apache Spark

Teradata

Hadoop

Job description

• We are currently seeking a Data Engineer experienced with cloud conversion/migration, with a preference for Teradata to AWS migrationThe Engineer must have Knowledge of Databricks.
• Must have demonstrable hands‑onexperience in creating/designing/implementing end‑to‑end data pipeline solutions. (Data ingestion, Data extraction, Data transformation, Data enrichment, Data loading, Data validation, Data processing, Data monitoring and maintenance, Data governance and security)
• In‑depth understanding of ETL and data ingestion processes used in large‑scale data warehouses

Position Overview:

The Data Engineer is a skilled professional responsible for creating end‑to‑end data pipelines, managing data ingestion, transformation, enrichment, loading, validation, monitoring, and maintenance processes. This role plays a crucial part in building and optimizing data architecture, ensuring seamless data flow between various systems, and supporting data‑driven decision‑making. The Data Engineer should possess expertise in cloud migration, Databricks, ETL processes, and data governance, particularly for large‑scale data warehouses.

Key Responsibilities:

• End‑to‑End Data Pipelines: Design, implement, and maintain robust data pipelines that cover the entire data lifecycle, from data ingestion to data loading, and ensure the efficient flow of data through the entire process.
• Data Ingestion: Develop and manage data ingestion processes to collect data from different sources, such as databases, APIs, logs, and other data repositories, ensuring data is collected in a reliable and scalable manner.
• Data Transformation: Perform data transformations and data enrichment to convert raw data into a structured and usable format, preparing it for storage and analysis.
• Data Loading: Implement data loading procedures to efficiently store processed data into data warehouses, data lakes, or other storage systems, ensuring data quality and integrity.
• Data Validation: Design and implement data validation mechanisms to verify data accuracy, consistency, and completeness, ensuring the data is reliable for analytical purposes.
• Data Processing: Utilize various data processing technologies and frameworks to handle large‑scale data volumes efficiently and effectively.
• Data Monitoring and Maintenance: Establish monitoring processes to ensure data pipelines run smoothly, and proactively address any issues that arise. Regularly maintain and optimize data pipelines for better performance and scalability.
• Cloud Migration: Plan and execute cloud migration strategies, transferring data and applications to cloud‑based platforms while ensuring security, scalability, and cost‑efficiency.
• Databricks Expertise: Leverage Databricks and Apache Spark to build scalable and high‑performance data processing workflows, providing insights and analytics on large datasets.
• ETL and Data Ingestion Processes: Understand and implement ETL (Extract, Transform, Load) processes used in large‑scale data warehouses, ensuring efficient data movement and integration.
• Data Governance and Security: Implement data governance best practices and security measures to protect sensitive data and ensure compliance with data privacy regulations.

Qualifications and Skills:

• Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.
• Proven experience as a Data Engineer or similar role with a strong focus on data pipeline development.
• Expertise in creating end‑to‑end data pipelines, including data ingestion, extraction, transformation, loading, and validation.
• Proficiency in cloud migration, particularly with cloud platforms like AWS, Azure, or Google Cloud.
• Experience with Databricks and Apache Spark for large‑scale data processing and analytics.
• In‑depth knowledge of ETL processes and best practices used in large‑scale data warehouses.
• Knowledge of big data platforms such as Hadoop, Spark, HBase, and Teradata
• SQL Databases: Oracle DB, DB2 - On‑Prem and AWS (preferred)
• Strong programming skills in languages like Python, SQL, or Java for data manipulation and scripting.
• Knowledge of distributed data storage systems, data lakes, and data warehousing concepts.
• Excellent problem‑solving and analytical skills, with an ability to handle complex data‑related challenges.
• Strong communication and collaboration skills to work effectively with cross‑functional teams.

Nice to have

• Precisely Connect
• Fivetran Transformations
• Prophecy

As a Data Engineer, the individual plays a pivotal role in designing and implementing robust data pipelines and facilitating smooth data integration, empowering organizations to harness the power of data for informed decision‑making and business growth.

Benefits (employee contribution):

• Health insurance
• Health savings account
• Vision insurance
• Flexible spending accounts
• Life insurance

All qualified applicants will receive consideration for employment without regard to age, race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.