Data Engineer - AWS

Inizio Partners Corp; Dallas (TX); On-site; USD 90,000 - 120,000; Full time

Job summary

A leading data engineering firm in Dallas is seeking an experienced Data Engineer proficient in AWS, Python, and SQL. In this role, you will design and implement data pipelines, ensuring the delivery of accurate and reliable data across the organization. The position requires strong experience in ETL processes and the ability to manage complex projects. A Bachelor's degree in a technical field is required for this role.

Qualifications

• 3+ years of experience with AWS tools such as S3 and EC2.
• 5+ years in data engineering and ETL using Python, SQL, and Databricks.
• Experience with data consistency and reliability.

Responsibilities

• Implement and manage AWS tools and environment.
• Build complex scripts in Python/Pyspark for data transformation.
• Work with users to develop value-adding datasets.

Skills

AWS Console

Python

SQL

Data pipeline development

ETL processes

Version control (GitHub)

Delta Lake

Education

Bachelor's degree in information systems, computer science, or related technical field

Tools

Databricks

Airflow

Job description

We are expanding our Data Engineering toolset and need an experienced AWS, Python and SQL Data Engineer. This role will be required to design and implement our new data pipelines using AWS, and train other team members. As a Data Engineer on our Data Engineering team, you will also design, write, scale, and maintain complex data pipelines using Python within our development framework. You will contribute to the organizations success by partnering with business, data science and data visualization team and transforming ingested data to meet the reporting requirements. Collaborating across disciplines, you will identify internal and external data sources to design pipelines, table structures, define ETL strategies and automate error-handling and validation. This team works with various stakeholders and divisions, including the executive team, with the goal of providing timely, accurate, and reliable data to thousands of users. Your role will be critical in defining the appropriate architecture and processes to support our Databricks data infrastructure, that is flexible, agile, reliable, responsive, and scalable. As a member of the Data Engineering team, you will report to the Manager of Data Engineering.

Primary Responsibilities

• Implement and manage AWS tools and environment.
• Using Python/ Pyspark in databricks, build and write complex scripts to transform ingested data to meet the business requirements.
• Work with internal and external users and providers to build datasets that add value to the business and allow for informed business decisions.
• Ensure data consistency, accuracy and reliability as data and business requirements change.

Required Skills

• 3+ years of AWS Console, S3, Airflow, EC2, EMR
• 5+ years of data engineering, data pipeline development, and ETL experience using Python, SQL, and databricks.
• Experience with using version control tools like GitHub.
• Experience with Delta lake, Unity Catalog, Delta Sharing, Delta Live Tables(DLT)
• Proficiency in the Python scripting language, SQL, Cloud databases, and ETL development processes & tools
• Ability to initiate, drive, and manage projects with competing priorities.
• Ability to communicate effectively with business leaders, IT leadership, and engineers.
• Must have a passion for data and helping the business turn data into information and action.

Required Education/Experience

• Bachelors degree in information systems, computer science, or related technical field