Data Engineer – Cloud Pipeline Development

The Elite Job; Genoa Township (IL); Hybrid; USD 110,000 - 150,000; Full time

Job summary

The Elite Job in the United States is seeking a seasoned Data Engineer – Cloud Pipeline Development to design, build, and maintain scalable cloud data pipelines that power BI, analytics, and reporting. You will work with cloud platforms, data integration, and large-scale processing to transform raw data into reliable insights.

You will collaborate with analysts, data scientists, engineers, and business stakeholders to ensure data accuracy, security, and accessibility, while optimizing

Qualifications

• Bachelors in CS/IT/related field; masters preferred but not required.
• Strong Python and SQL for data processing and automation.
• Experience with cloud platforms (AWS/Azure/GCP) and ETL/ELT patterns.

Responsibilities

• Design, build, deploy, and maintain scalable cloud data pipelines.
• Integrate data from multiple sources and develop ETL/ELT workflows.
• Collaborate with analysts, data scientists, software engineers, and stakeholders to ensure data quality.
• Monitor performance, optimize queries, and ensure secure data delivery.

Skills

Python

SQL

ETL/ELT

Cloud platforms

Data warehousing

Airflow

Data modeling

JSON/Parquet

APIs

Testing

Education

Bachelor's degree in CS/IT/related field

Master's degree is an advantage

Certifications may be considered

Tools

Git/GitHub

AWS

Azure

GCP

Airflow

Job description

1. Job Summary

The Elite Job is seeking a skilled and motivated Data Engineer – Cloud Pipeline Development to design, develop, implement, and maintain reliable data pipelines that support business intelligence, analytics, reporting, and data-driven decision-making. The ideal candidate will have a strong understanding of data engineering principles, cloud-based data platforms, data integration, and large-scale data processing.

In this role, the Data Engineer will be responsible for building scalable and efficient cloud data pipelines that collect, transform, validate, and deliver data from multiple sources to centralized data platforms. The successful candidate will collaborate with data analysts, data scientists, software developers, cloud architects, and business stakeholders to ensure that data is accessible, accurate, secure, and ready for analytical use.

The position requires a combination of programming expertise, database knowledge, cloud technology experience, and problem-solving skills. The candidate should be comfortable working with structured and unstructured data, automating data workflows, optimizing pipeline performance, and troubleshooting complex data processing issues.

At The Elite Job, we value innovation, technical excellence, collaboration, and continuous learning. This opportunity is suitable for professionals who want to contribute to modern cloud data infrastructure and help organizations transform raw data into meaningful business insights.

2. Key Responsibilities

• Cloud Data Pipeline Development: Design, build, deploy, and maintain scalable data pipelines using cloud-based infrastructure and modern data engineering technologies.
• Data Integration: Extract and integrate data from multiple sources, including relational databases, APIs, enterprise applications, cloud storage, and third-party systems.
• ETL and ELT Processes: Develop efficient Extract, Transform, Load (ETL) and Extract, Load, Transform (ELT) workflows to prepare data for reporting, analytics, and business intelligence.
• Cloud Platform Management: Work with platforms such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP) to develop and manage data processing solutions.
• Data Transformation: Apply business rules, data cleansing techniques, validation logic, and transformation procedures to ensure data consistency and quality.
• Data Warehouse Development: Support the design and implementation of cloud data warehouses, data lakes, and lakehouse architectures that meet organizational requirements.
• Workflow Orchestration: Implement and maintain automated data workflows using orchestration tools such as Apache Airflow, AWS Step Functions, Azure Data Factory, or comparable technologies.
• Programming and Automation: Develop reusable scripts, utilities, and data processing components using Python, SQL, and other relevant programming languages.
• Performance Optimization: Monitor pipeline execution, identify bottlenecks, optimize queries, and improve processing speed, resource utilization, and operational efficiency.
• Data Quality Management: Establish data validation checks, reconciliation procedures, and monitoring mechanisms to identify missing, duplicate, inconsistent, or inaccurate records.
• Real-Time Data Processing: Develop batch and streaming data pipelines where required, using technologies such as Apache Kafka, Amazon Kinesis, or equivalent services.
• Security and Compliance: Apply appropriate access controls, encryption, secrets management, and data protection practices to safeguard sensitive business information.
• Monitoring and Troubleshooting: Investigate pipeline failures, resolve data processing errors, implement alerts, and maintain reliable data delivery.
• Documentation: Prepare technical documentation covering pipeline architecture, data models, dependencies, operational procedures, and troubleshooting guidelines.
• Cross-Functional Collaboration: Work closely with analytics, engineering, product, and business teams to understand requirements and deliver dependable data solutions.
• Testing and Deployment: Develop unit tests, integration tests, and data validation procedures while supporting controlled deployments across development, testing, and production environments.
• Continuous Improvement: Evaluate emerging cloud technologies and data engineering practices to improve scalability, reliability, maintainability, and cost efficiency.

3. Required Skills and Qualifications

Educational Qualifications

• Bachelors degree in Computer Science, Information Technology, Software Engineering, Data Engineering, Computer Engineering, or a related technical field.
• A masters degree in a relevant discipline is an advantage but is not mandatory.
• Equivalent practical experience and relevant technical certifications may also be considered.

Technical Skills

• Programming: Strong proficiency in Python and SQL for data processing, automation, querying, and transformation.
• Cloud Computing: Practical knowledge of at least one major cloud platform, such as AWS, Microsoft Azure, or Google Cloud Platform.
• Data Engineering: Understanding of ETL/ELT architecture, data integration patterns, batch processing, and distributed data processing.
• Database Technologies: Experience with relational databases such as PostgreSQL, MySQL, SQL Server, or similar systems.
• Cloud Data Services: Familiarity with services such as Amazon S3, AWS Glue, Amazon Redshift, Azure Data Factory, Azure Data Lake Storage, Google BigQuery, or equivalent platforms.
• Data Warehousing: Knowledge of dimensional modeling, star schemas, snowflake schemas, fact tables, dimension tables, and analytical data structures.
• Big Data Technologies: Familiarity with Apache Spark, PySpark, Hadoop, or comparable distributed processing frameworks is beneficial.
• Workflow Automation: Understanding of scheduling, dependency management, orchestration, retries, and failure recovery.
• Version Control: Experience using Git and collaborative development platforms such as GitHub, GitLab, or Bitbucket.
• Data Formats: Understanding of CSV, JSON, XML, Parquet, Avro, and other common data interchange formats.
• API Integration: Ability to integrate REST APIs and handle authentication, pagination, rate limits, and error responses.
• Testing and Quality Assurance: Knowledge of automated testing, data validation, and pipeline quality assurance practices.
• Security Awareness: Understanding of data privacy, identity and access management, encryption, and secure credential handling.

Professional Qualifications

• Strong analytical thinking and problem-solving abilities.
• Excellent attention to detail and commitment to data accuracy.
• Ability to communicate technical concepts clearly to technical and non-technical stakeholders.
• Strong organizational skills and the ability to manage multiple tasks.
• Willingness to learn new technologies and adapt to evolving business requirements.
• Ability to work independently while contributing effectively to a collaborative team.

4. Experience

The required experience will depend on the complexity of the assigned projects and the level of the position.

• Entry-Level Candidates: Candidates with relevant academic projects, internships, cloud training, or practical programming experience may be considered for junior opportunities.
• Intermediate Candidates: Approximately 2–5 years of experience in data engineering, cloud development, ETL/ELT development, or a related technical role is preferred.
• Senior-Level Candidates: Extensive experience designing production-grade cloud data pipelines, managing complex integrations, and optimizing data platforms will be advantageous for senior responsibilities.

Candidates should ideally demonstrate practical experience building data pipelines, working with databases, writing SQL queries, automating data transformations, and troubleshooting data-related issues.

Experience with cloud-native data services, infrastructure automation, CI/CD pipelines, data observability, and production monitoring is an advantage.

A portfolio of relevant projects, technical demonstrations, or certifications can strengthen an application. Relevant certifications may include AWS Certified Data Engineer – Associate, Microsoft Certified: Fabric Data Engineer Associate, or Google Cloud Professional Data Engineer, depending on the candidates experience and the technologies used.

5. Working Hours

The working arrangements for the Data Engineer – Cloud Pipeline Development position will be confirmed during the recruitment process.

Employment Type: Full-time, subject to the specific vacancy.

Work Arrangement: Remote, hybrid, or office-based, depending on the role and project requirements.

Working Schedule: Standard business hours or an agreed schedule aligned with team and client requirements.

Flexibility: Flexible working arrangements may be available where operational requirements permit.

Team Collaboration: Employees may need to attend virtual meetings, technical reviews, project planning sessions, and cross-functional discussions.

Production Support: Occasional support outside standard working hours may be required for critical deployments, pipeline failures, or scheduled maintenance, subject to the applicable employment arrangements.

Candidates should be comfortable managing project deadlines, coordinating with distributed teams, and maintaining consistent communication throughout the development lifecycle.

Specific working hours, time-zone expectations, overtime arrangements, and remote-work eligibility will be communicated for the confirmed position.

6. Knowledge, Skills, and Abilities

Knowledge

• Understanding of data engineering principles, modern data architectures, and cloud-native technologies.
• Knowledge of relational database concepts, SQL optimization, indexing, and data modeling.
• Understanding of ETL/ELT workflows and data lifecycle management.
• Familiarity with data lakes, data warehouses, lakehouses, and analytical data platforms.
• Awareness of data governance, metadata management, data lineage, and data quality standards.
• Understanding of cloud security, resource management, scalability, and cost optimization.
• Familiarity with batch processing, streaming architectures, and distributed computing concepts.

Skills

• Ability to develop maintainable and reusable data pipeline components.
• Strong SQL development and data transformation skills.
• Ability to automate repetitive data engineering tasks using Python or similar programming tools.
• Competence in identifying, investigating, and resolving pipeline failures.
• Ability to optimize data processing workflows for performance and cost.
• Effective technical documentation and communication skills.
• Ability to use version control, testing frameworks, and deployment tools.
• Strong time management, prioritization, and project coordination skills.

Abilities

• Ability to analyze complex data requirements and translate them into practical technical solutions.
• Ability to manage large datasets and develop scalable data processing workflows.
• Ability to identify data quality issues and implement preventive controls.
• Ability to adapt to new tools, frameworks, and cloud technologies.
• Ability to work collaboratively with cross-functional and geographically distributed teams.
• Ability to take ownership of assigned tasks and deliver reliable results.
• Ability to balance performance, security, maintainability, and cost when designing data solutions.
• Ability to learn continuously and contribute to improvements in engineering standards and operational processes.

7. Benefits

• Competitive Compensation: Salary aligned with the candidates experience, technical skills, and responsibilities.
• Flexible Work Options: Remote or hybrid working opportunities where offered for the specific vacancy.
• Professional Development: Opportunities to strengthen skills in cloud computing, data engineering, automation, and distributed processing.
• Certification Support: Access to relevant learning resources and potential certification assistance where available.
• Career Growth: Opportunities to progress into senior data engineering, cloud architecture, data platform engineering, or technical leadership roles.
• Collaborative Environment: Opportunities to work with professionals across engineering, analytics, software development, and business teams.
• Modern Technology Exposure: Potential experience with cloud-native data platforms, automation tools, and contemporary data architectures.
• Meaningful Technical Work: Opportunities to contribute to reliable data infrastructure that supports reporting, analytics, and business decisions.
• Performance Recognition: Recognition for quality work, technical contributions, and achievement of agreed objectives, subject to company policy.
• Work-Life Balance: Reasonable workload planning and flexible arrangements where supported by the position.
• Actual salary, insurance, paid leave, bonuses, equipment, training budgets, and other benefits will be specified in the official vacancy details or employment offer. These benefits should not be assumed unless confirmed by the employer.

8. Why Join The Elite Job?

Joining The Elite Job for the Data Engineer – Cloud Pipeline Development opportunity offers the potential to contribute to modern data infrastructure and develop valuable technical expertise in cloud-based engineering.

This role is suited to professionals who enjoy solving complex technical problems, building dependable systems, and making data more accessible to the people who need it. It provides an opportunity to apply programming, database, and cloud computing skills to practical data engineering challenges.

As a Data Engineer, you may have opportunities to:

• Develop expertise in cloud-based data pipeline architecture and automation.
• Work on data integration and transformation challenges involving multiple systems.
• Strengthen your knowledge of data warehousing, data lakes, and modern analytical platforms.
• Improve your technical capabilities through practical problem-solving and continuous learning.
• Collaborate with professionals across different technical and business functions.
• Build a career path toward senior data engineering, cloud data architecture, data platform development, or engineering leadership.
• Contribute to better data reliability, processing efficiency, and business intelligence capabilities.

The Elite Job welcomes candidates who demonstrate technical curiosity, accountability, attention to detail, and a commitment to professional growth.

If you are passionate about cloud technologies, data processing, and building scalable solutions, this position may be an excellent opportunity to develop your career in data engineering.