Data Engineering-ETL-Big Data-GCP-Java

BCforward; Hyderabad; On-site; INR 1,400,000 - 2,000,000; Full time

Job summary

BCforward seeks a skilled Data Engineer to build real-time data pipelines on Google Cloud Platform. You will implement scalable data solutions, write robust Java-based services, and ensure reliable data processing for analytics use cases.

Responsibilities include designing ETL workflows, working with BigQuery, Spark/Hadoop ecosystems, and collaborating with cross-functional teams to meet business needs.

Qualifications

• Experience building real-time data pipelines using GCP.
• Strong Java development skills and backend APIs design.
• Experience with Spark/Hadoop ecosystems and data warehousing.
• Proficiency in SQL analytics and data modeling.

Responsibilities

• Design and implement Big Data pipelines on GCP.
• Develop REST APIs and data services with Java.
• Collaborate with stakeholders to deliver scalable data solutions.

Skills

Big Data

ETL - Big Data / Data Warehousing

GCP

Java

SQL

Tools

BigQuery

Dataproc

GKE

Cloud Dataflow

Cloud Functions

Job description

Preferred Skills: 6+ years relevant experience into Big Data; ETL - Big Data / Data Warehousing; GCP; Java

Skills:Big Data; ETL - Big Data / Data Warehousing; GCP; Java

We are looking for a highly skilled Engineer with a solid experience of building Bigdata, GCP Cloud based real time data pipelines and REST APIs with Java frameworks. The Engineer will play a crucial role in designing, implementing, and optimizing data solutions to support our organizations data-driven initiatives. This role requires expertise in data engineering, strong problem-solving abilities, and a collaborative mindset to work effectively with various stakeholders.

This role will be focused on the delivery of innovative solutions to satisfy the needs of our business. As an agile team we work closely with our business partners to understand what they require, and we strive to continuously improve as a team.

Technical Skills

1. Core Data Engineering Skills Proficiency in using GCPs big data tools like: BigQuery: For data warehousing and SQL analytics. Dataproc: For running Spark and Hadoop clusters. GCP Dataflow: For stream and batch data processing.(High level Idea)GCP Pub/Sub: For real-time messaging and event ingestion.(High level Idea) Expertise in building automated, scalable, and reliable pipelines using custom Python/Scala solutions or Cloud Data Functions.

2. Programming and Scripting Strong coding skills in SQL, and Java. Familiarity with APIs and SDKs for GCP services to build custom data solutions.

3. Cloud Infrastructure Understanding of GCP services such as Cloud Storage, Compute Engine, and Cloud Functions. Familiarity with Kubernetes (GKE)and containerization for deploying data pipelines. (Optional but Good to have)

4. DevOps and CI/CD Experience setting up CI/CD pipelines using Cloud Build, GitHub Actions, or other tools. Monitoring and logging tools like Cloud Monitoringand Cloud Logging for production workflows.

5. Backend Development (Spring Boot & Java)-; Design and develop RESTful APIs and microservices using Spring Boot. Implement business logic, security, authentication (JWT/OAuth), and database operations. Work with relational databases (MySQL, PostgreSQL, MongoDB, Cloud SQL). Optimize backend performance, scalability, and maintainability. Implement unit testing and integration testing.

Soft Skills

1. Innovation and Problem-Solving Ability to think creatively and design innovative solutions for complex data challenges. Experience in prototyping and experimenting with cutting-edge GCP tools or third-party integrations. Strong analytical mindset to transform raw data into actionable insights.

2. Collaboration Teamwork: Ability to collaborate effectively with data analysts, and business stakeholders. Communication: Strong verbal and written communication skills to explain technical concepts to non-technical audiences.

3. Adaptability and Continuous Learning Open to exploring new GCP features and rapidly adapting to changes in cloud technology.