Cloud Data Engineer – Lakehouse Pipeline Development
The Elite Job
Get a reply from this employer
Your resume and cover letter, tailored to exactly what they’re hiring for.
Benefits offered by this job
Job summary
The Elite Job is seeking a skilled Cloud Data Engineer – Lakehouse Pipeline Development to design, develop, implement, and maintain scalable cloud-based data pipelines and lakehouse architectures. This role focuses on building robust ETL/ELT pipelines, integrating data from multiple sources, and enabling analytics-ready data assets across cloud platforms.
You will collaborate with data architects, analytics engineers, data scientists, and software developers to ensure performance, security, and
Qualifications
- Bachelor's degree in Computer Science, IT, Software Engineering, Data Engineering, or related field.
- Masters degree advantageous but not mandatory.
- Relevant cloud/data engineering certifications are beneficial.
Responsibilities
- Design, develop, and maintain lakehouse solutions for scalable data architectures.
- Build, deploy, and optimize batch and streaming data pipelines.
- Collaborate with data architects, analysts, data scientists, and engineers.
- Ensure data quality, governance, and security across data platforms.
Skills
Education
Tools
Job description
The Elite Job is seeking a skilled and detail-oriented Cloud Data Engineer – Lakehouse Pipeline Development to design, develop, implement, and maintain scalable cloud-based data pipelines and modern lakehouse architectures. This role is ideal for a technology professional who enjoys working with large datasets, building reliable data processing solutions, and transforming complex data into high-quality, analytics-ready assets. The successful candidate will be responsible for developing robust ETL/ELT pipelines, integrating data from multiple sources, and implementing efficient data storage and processing solutions across cloud platforms. The role involves working with data lakes, data warehouses, distributed processing frameworks, and lakehouse technologies to support business intelligence, advanced analytics, machine learning, and data-driven decision-making. The Cloud Data Engineer will collaborate closely with data architects, analytics engineers, data scientists, software developers, and business stakeholders to ensure that data solutions meet performance, security, reliability, and business requirements. The ideal candidate combines strong programming skills with an understanding of cloud infrastructure, data modelling, data governance, and modern engineering practices. This position offers an opportunity to contribute to innovative data engineering initiatives, improve enterprise data capabilities, and develop expertise in next-generation cloud data platforms.
- Lakehouse Architecture Development: Design, develop, and maintain scalable lakehouse solutions that combine the flexibility of data lakes with the reliability and management capabilities of data warehouses.
- Data Pipeline Engineering: Build, deploy, and optimize batch and streaming data pipelines to ingest, transform, validate, and deliver structured, semi-structured, and unstructured data.
- ETL/ELT Development: Implement efficient extraction, transformation, and loading processes using modern data engineering frameworks, reusable components, and automated workflows.
- Cloud Platform Integration: Develop data solutions using cloud services such as AWS, Microsoft Azure, or Google Cloud Platform, depending on project requirements.
- Distributed Data Processing: Use technologies such as Apache Spark, PySpark, SQL, and related processing frameworks to handle large-scale datasets efficiently.
- Lakehouse Table Management: Work with open table formats and storage technologies such as Delta Lake, Apache Iceberg, or Apache Hudi to support reliable data updates, schema evolution, and transaction management.
- Data Source Integration: Connect data pipelines to databases, APIs, enterprise applications, event streams, file systems, and third-party data sources.
- Data Modelling and Transformation: Develop efficient data models, transformation logic, and curated datasets that support reporting, analytics, and downstream applications.
- Data Quality Assurance: Implement validation rules, reconciliation checks, schema enforcement, anomaly detection, and monitoring mechanisms to maintain data accuracy and consistency.
- Pipeline Orchestration: Schedule, coordinate, and monitor data workflows using tools such as Apache Airflow, cloud-native orchestration services, or equivalent platforms.
- Performance Optimization: Identify bottlenecks in data processing, storage, and query execution, and optimize partitioning, file sizes, caching, and resource utilization.
- Security and Governance: Apply appropriate access controls, encryption, data classification, auditing, and governance practices to protect sensitive information.
- Monitoring and Troubleshooting: Investigate pipeline failures, resolve data processing issues, manage dependencies, and implement preventive measures to improve operational reliability.
- CI/CD and Automation: Establish automated testing, deployment pipelines, infrastructure configuration, and version control practices to support consistent and repeatable releases.
- Documentation: Maintain technical documentation covering pipeline architecture, data flows, transformation logic, dependencies, operational procedures, and troubleshooting guidelines.
- Cross-Functional Collaboration: Work with data analysts, business intelligence teams, data scientists, and engineering stakeholders to understand requirements and deliver dependable data products.
- Continuous Improvement: Evaluate emerging cloud data technologies, engineering methodologies, and lakehouse capabilities to improve scalability, efficiency, maintainability, and cost effectiveness.
- Bachelors degree in Computer Science, Information Technology, Software Engineering, Data Engineering, or a related technical discipline.
- A masters degree in a relevant field is an advantage but is not mandatory.
- Relevant professional certifications in cloud computing, data engineering, or data analytics are beneficial.
- Strong proficiency in SQL, including complex queries, joins, aggregations, window functions, and query optimization.
- Practical programming experience with Python, PySpark, Scala, or another suitable data engineering language.
- Understanding of data lake, data warehouse, and lakehouse architectures.
- Experience developing and maintaining ETL/ELT pipelines for large-scale data processing.
- Knowledge of cloud data services available on AWS, Microsoft Azure, or Google Cloud Platform.
- Familiarity with Apache Spark and distributed computing concepts.
- Understanding of Delta Lake, Apache Iceberg, Apache Hudi, or comparable lakehouse technologies.
- Experience with relational databases and familiarity with NoSQL databases or other modern data stores.
- Knowledge of data modelling techniques, including dimensional modelling, star schemas, and slowly changing dimensions.
- Understanding of file formats such as Parquet, Avro, JSON, and CSV.
- Familiarity with workflow orchestration tools and automated scheduling.
- Knowledge of Git, source control, code reviews, and collaborative development practices.
- Understanding of REST APIs, data ingestion patterns, and event-driven data processing.
- Familiarity with unit testing, integration testing, data validation, and automated deployment practices.
- Awareness of data privacy, information security, access management, and governance standards.
- Strong analytical and problem-solving abilities.
- Excellent attention to detail and a commitment to data quality.
- Ability to translate business requirements into technical data solutions.
- Effective written and verbal communication skills.
- Ability to work independently and collaborate with multidisciplinary teams.
- Strong organizational skills and the ability to manage competing priorities.
- Willingness to learn new technologies and adapt to evolving data engineering practices.
- Typically, 2–5 years of relevant experience in data engineering, cloud data platforms, ETL/ELT development, or large-scale data processing is preferred.
- Demonstrated experience building, deploying, and maintaining production data pipelines.
- Practical exposure to at least one major cloud platform, such as AWS, Azure, or Google Cloud.
- Experience with Python, SQL, and distributed data processing tools.
- Familiarity with lakehouse architecture and modern table formats is highly desirable.
- Experience implementing data quality checks, monitoring, logging, and pipeline recovery mechanisms.
- Understanding of performance optimization, scalable data storage, and cloud resource management.
- Experience working in Agile development environments and using version control and CI/CD tools is an advantage.
- Candidates with strong foundational knowledge, relevant projects, internships, or demonstrable practical experience may also be considered for junior-level opportunities, depending on the specific vacancy requirements.
- Employment Type: Full-time.
- Work Arrangement: Remote, hybrid, or office-based, depending on the position and business requirements.
- Typical Schedule: Approximately 40 hours per week, subject to the applicable employment agreement.
- Working Days: Generally Monday to Friday, with schedules determined by the assigned team and location.
- Collaboration: Occasional flexibility may be required to coordinate with colleagues or stakeholders across different time zones.
- Project Deadlines: Some additional flexibility may be necessary during major deployments, production incidents, or critical project milestones.
- The final working schedule, location requirements, and employment conditions will be communicated during the recruitment process.
The candidate should understand the principles of data ingestion, transformation, storage, processing, orchestration, and consumption. They should be able to design reliable data workflows that accommodate growing data volumes while maintaining data integrity and operational efficiency.
The candidate should understand cloud storage, compute resources, identity and access management, networking fundamentals, monitoring, and resource optimization. The ability to select appropriate cloud services based on technical and business requirements is important.
The candidate should be able to contribute to lakehouse implementation, including raw and curated data layers, schema management, incremental processing, data versioning, and reliable data access. Familiarity with concepts such as ACID transactions, schema evolution, and time travel in supported table formats is beneficial.
The candidate should be capable of writing readable, maintainable, and reusable code. They should understand error handling, logging, testing, modular development, and automation practices that improve the reliability of data pipelines.
The candidate must be able to investigate data discrepancies, identify root causes, interpret processing failures, and recommend effective solutions. Strong reasoning skills are essential when working with complex datasets and interconnected systems.
The ability to explain technical concepts clearly, document solutions, participate in design discussions, and coordinate with cross-functional teams is important for successful project delivery.
The candidate should understand secure data handling, least-privilege access, encryption, auditability, and responsible data management. They should follow applicable organizational policies and relevant regulatory requirements.
The candidate should demonstrate a willingness to explore new tools, improve existing systems, share technical knowledge, and respond constructively to changing business priorities.
- Competitive Compensation: Salary packages aligned with experience, technical expertise, responsibilities, and market conditions.
- Flexible Working Options: Remote or hybrid working arrangements where available and approved for the role.
- Professional Development: Opportunities to strengthen skills in cloud engineering, distributed computing, data architecture, and lakehouse technologies.
- Certification Support: Potential access to approved training programmes and relevant professional certification resources.
- Career Advancement: Opportunities to progress into senior data engineering, cloud architecture, platform engineering, or technical leadership roles.
- Collaborative Environment: Opportunities to work with technology professionals across data, software, analytics, and cloud infrastructure teams.
- Technical Learning: Exposure to modern engineering tools, scalable architectures, and emerging data platform capabilities.
- Performance Recognition: Opportunities for recognition based on contribution, quality of delivery, collaboration, and achievement of agreed objectives.
- Work-Life Balance: Support for sustainable working practices, clear priorities, and effective workload management.
- Employee Benefits: Any applicable paid leave, insurance, retirement contributions, or other benefits will depend on the employment contract, location, and company policy.
- Specific benefits, compensation, and eligibility conditions will be confirmed by the hiring team during the recruitment process.
- Joining The Elite Job as a Cloud Data Engineer – Lakehouse Pipeline Development offers an opportunity to contribute to modern data engineering solutions and build a career in a rapidly evolving technology field.
- Work on Modern Data Technologies: Apply your skills to cloud-native data pipelines, distributed processing, lakehouse architectures, and enterprise data integration.
- Solve Meaningful Technical Challenges: Help develop reliable and scalable data solutions that improve information accessibility, support analytics, and enable better business decisions.
- Expand Your Technical Expertise: Strengthen your knowledge of cloud platforms, automation, data quality, data governance, and performance optimization.
- Collaborate Across Teams: Work with professionals who bring different perspectives to data architecture, analytics, software engineering, and business problem-solving.
- Build a Long-Term Career: Develop the technical and professional capabilities required to pursue future opportunities in senior data engineering, cloud architecture, data platform development, and technical leadership.
- Contribute to Engineering Excellence: Help establish reusable pipeline frameworks, reliable operational processes, and maintainable data solutions that can evolve with business needs.
- If you enjoy solving complex data problems, building dependable systems, and continuously improving your engineering capabilities, this role may be an excellent next step in your professional journey.
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
