QX Impact - Senior Data Engineer - ETL/Python

QX Impact; Tirupati; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

QX Impact in Tirupati is seeking a Senior Data Engineer to design and maintain robust data pipelines supporting analytics, data science, and BI solutions. You will collaborate with data scientists, engineers, and UX developers to build scalable data architectures and deliver reliable data products.

You will work with enterprise data platforms, optimize cloud infrastructure on AWS/Azure, and implement modern ETL, data warehousing, and streaming pipelines to empower decision making across the

Qualifications

• Bachelor’s degree in CS/Math/Engineering or related field.
• 5+ years in enterprise data platforms and data lakes.
• 3-5 years designing and implementing data warehouses.
• Strong SQL with stored procedures.
• Proficient in Spark (Python/Scala) and Spark Streaming.
• Experience with AWS or Azure data services.
• Familiar with Kafka, Spark, or Flink.
• Hands-on with Airflow or Prefect.
• Knowledge of CI/CD and Git/Jenkins.
• Strong problem-solving and collaboration.

Responsibilities

• Design, develop, test, and maintain data pipelines and ETL architectures.
• Map out data systems and define integrations, ETL, BI, and AI processes.
• Prepare and optimize data for predictive and prescriptive modeling.
• Collaborate with teams to integrate ERP data into the enterprise data lake.
• Enhance cloud data infrastructure on AWS/Azure for scalability and performance.
• Utilize big data tools to optimize data acquisition and preparation.
• Build architectures to move data to/from data lakes and data warehouses.
• Develop and curate data models for analytics, dashboards, and reports.
• Conduct code reviews and maintain production-level code.
• Monitor, troubleshoot, and resolve data ingestion workflows.
• Drive innovation and implement efficient new approaches.

Skills

SQL

Spark (Python/Scala)

Spark Streaming

AWS

Azure

Data Lakes

Data Warehousing

Airflow

Prefect

CI/CD

Git

Jenkins

Problem-solving

Communication

Collaboration

Education

Bachelor's degree in Computer Science/Mathematics/Engineering

Tools

AWS Glue

Azure Data Factory

Redshift

Snowflake

Apache Kafka

Apache Spark

Flink

Apache Airflow

Prefect

Git

Jenkins

Job description

Job Summary We are looking for a Senior Data Engineer who is creative, collaborative, and adaptable to join our agile team of data scientists, engineers, and UX developers. The role focuses on building and maintaining robust data pipelines to support advanced analytics, data science, and BI solutions.

Job Summary We are looking for a Senior Data Engineer who is creative, collaborative, and adaptable to join our agile team of data scientists, engineers, and UX developers. The role focuses on building and maintaining robust data pipelines to support advanced analytics, data science, and BI solutions. As a Senior Data Engineer, you will work with internal and external data, collaborate with data scientists, and contribute to the design, development, and deployment of innovative solutions.

Key Responsibilities

• Design, develop, test, and maintain optimal data pipeline and ETL architectures.
• Map out data systems and define/design required integrations, ETL, BI, and AI systems/processes.
• Prepare and optimize data for predictive and prescriptive modeling.
• Collaborate with teams to integrate ERP data into the enterprise data lake, ensuring seamless flow and quality.
• Enhance cloud data infrastructure on AWS or Azure for scalability and performance.
• Utilize big data tools and frameworks to optimize data acquisition and preparation.
• Build architectures to move data to/from data lakes and data warehouses for advanced analytics.
• Develop and curate data models for analytics, dashboards, and reports.
• Conduct code reviews, maintain production-level code, and implement testing approaches.
• Monitor, troubleshoot, and resolve data ingestion workflows to maintain reliability and uptime.
• Drive innovation and implement efficient new approaches to data engineering tasks.

Must-Have Skills

• Bachelor's degree in Computer Science, Mathematics, Engineering, or a related field.
• 5+ years of experience working with enterprise data platforms, including building and managing data lakes.
• 3 - 5 years of experience designing and implementing data warehouse solutions.
• Expertise in SQL, including developing stored procedures (SP) and applying advanced data design concepts.
• Proficiency in Spark (Python/Scala) and Spark Streaming for real-time data pipelines.
• Experience with AWS or Azure services (e.g., AWS Glue, Azure Data Factory, Redshift, Snowflake).
• Familiarity with big data tools such as Apache Kafka, Apache Spark, or Flink.
• Hands-on experience with orchestration tools (e.g., Apache Airflow, Prefect).
• Knowledge of CI/CD processes, version control (e.g., Git, Jenkins), and deployment automation.
• Strong problem-solving, communication, and collaboration skills.

Good-to-Have Skills

• Experience in integrating ERP data into data lakes.
• Experience with traditional ETL tools (e.g., Talend, Pentaho).

Competencies

• Tech Savvy: Anticipating and adopting innovations in business-building digital and technology applications.
• Self-Development: Actively seeking new ways to grow and be challenged using both formal and informal development channels.
• Action Oriented: Taking on new opportunities and tough challenges with a sense of urgency, high energy, and enthusiasm.
• Customer Focus: Building strong customer relationships and delivering customer-centric solutions.
• Optimize Work Processes: Knowing the most effective and efficient processes to get things done, with a focus on continuous improvement.

Why Join Us ?

• Be part of a collaborative and agile team driving cutting-edge AI and data engineering solutions.
• Work on impactful projects that make a difference across industries.
• Opportunities for professional growth and continuous learning.
• Competitive salary and benefits package.

(ref:hirist.tech)