Design, develop, and maintain scalable batch and real-time data pipelines using Python and PySpark. Build cloud-based data processing solutions, CI/CD pipelines, and streaming systems while ensuring data quality, performance, and scalability. Collaborate with cross-functional teams to translate business requirements into reliable data architectures. The role involves technologies including Kafka, Flink, Kubernetes, Hadoop, Spark, MongoDB, and cloud platforms such as Azure, AWS, or GCP.
Job Summary:
Key Responsibilities:
We are looking for a skilled Big Data Engineer with strong experience in PySpark and data engineering. The ideal candidate will have hands-on experience in building scalable data pipelines, working with streaming data, and cloud platforms. Experience with Palantir Foundry is highly preferred
Experience Required: 5 – 10 years
- Design, develop, and maintain ETL/data pipelines using Python and PySpark
- Work with real-time/streaming data systems
- Build and optimize data processing solutions on cloud platforms
- Collaborate with cross-functional teams to understand business requirements
- Develop and maintain CI/CD pipelines
- Ensure data quality, performance, and scalability
- 5–10+ years of experience in Data Engineering
- Strong expertise in Python and PySpark
- Experience with Kafka, Flink, and Kubernetes
- Hands-on with Big Data technologies (Hadoop, Spark, MongoDB)
- Experience with cloud platforms (Azure, AWS, or GCP)
- Good understanding of data architecture and pipeline design
- Experience with Palantir Foundry
- Knowledge of Java
- Experience in consulting or Utilities domain
Similar Jobs
Software
Design, implement, and lead large-scale generative AI and ML systems (LLM fine-tuning, RAG, agents, Graph ML). Build REST APIs, deploy with Docker/Kubernetes, optimize big-data workflows, mentor engineers, and operate ML services on cloud platforms.
Top Skills:
Anomaly DetectionAWSAzureBig DataCi/CdData MiningDockerDocument DatabasesFastapiGenerative AiGCPGraph MlKnowledge BasesKubernetesLlmsNoSQLPythonPyTorchRagRestful ApisSagemakerTensorrtTriton
Big Data • Fintech • Information Technology • Insurance • Financial Services
Develop and maintain Java microservices and APIs using Spring Boot, integrate AWS services, manage cloud deployments, maintain CI/CD pipelines, troubleshoot performance issues, and collaborate with DevOps, QA, and business teams. The role also involves documenting architecture and deployment processes while ensuring application scalability and security.
Top Skills:
APIsAWSCi/CdCloud InfrastructureJavaMicroservicesSpring Boot
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Design, develop, test, and deploy cloud-native applications, APIs, microservices, integrations, and reusable platform components. Collaborate across Product, Architecture, Security, and Engineering teams to deliver scalable, secure solutions. Implement automated testing, CI/CD, monitoring, observability, and modernization initiatives using Azure and enterprise technologies. Contribute to architecture, documentation, code reviews, engineering standards, and AI-assisted development improvements.
Top Skills:
.NetAgileAi-Assisted Development ToolsAPIsApplication Performance OptimizationAutomated TestingAzureCi/CdCloud-Native Application DevelopmentJavaMicroservicesMonitoringObservabilitySource Control
What you need to know about the Hyderabad Tech Scene
Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.
