Design, develop, and maintain scalable batch and real-time data pipelines using Python and PySpark. Build cloud-based data processing solutions, CI/CD pipelines, and streaming systems while ensuring data quality, performance, and scalability. Collaborate with cross-functional teams to translate business requirements into reliable data architectures. The role involves technologies including Kafka, Flink, Kubernetes, Hadoop, Spark, MongoDB, and cloud platforms such as Azure, AWS, or GCP.
Job Summary:
Key Responsibilities:
We are looking for a skilled Big Data Engineer with strong experience in PySpark and data engineering. The ideal candidate will have hands-on experience in building scalable data pipelines, working with streaming data, and cloud platforms. Experience with Palantir Foundry is highly preferred
Experience Required: 5 – 10 years
- Design, develop, and maintain ETL/data pipelines using Python and PySpark
- Work with real-time/streaming data systems
- Build and optimize data processing solutions on cloud platforms
- Collaborate with cross-functional teams to understand business requirements
- Develop and maintain CI/CD pipelines
- Ensure data quality, performance, and scalability
- 5–10+ years of experience in Data Engineering
- Strong expertise in Python and PySpark
- Experience with Kafka, Flink, and Kubernetes
- Hands-on with Big Data technologies (Hadoop, Spark, MongoDB)
- Experience with cloud platforms (Azure, AWS, or GCP)
- Good understanding of data architecture and pipeline design
- Experience with Palantir Foundry
- Knowledge of Java
- Experience in consulting or Utilities domain
Similar Jobs
Software
Design, implement, and lead large-scale generative AI and ML systems (LLM fine-tuning, RAG, agents, Graph ML). Build REST APIs, deploy with Docker/Kubernetes, optimize big-data workflows, mentor engineers, and operate ML services on cloud platforms.
Top Skills:
Anomaly DetectionAWSAzureBig DataCi/CdData MiningDockerDocument DatabasesFastapiGenerative AiGCPGraph MlKnowledge BasesKubernetesLlmsNoSQLPythonPyTorchRagRestful ApisSagemakerTensorrtTriton
Fintech • Mobile • Payments • Software • Financial Services
Conduct customer and business onboarding through CDD and EDD reviews, assess identity documents, ownership structures, source of wealth, adverse media, and transaction activity. Maintain accurate KYC records, perform risk assessments, identify money laundering and fraud indicators, escalate risks, and file Suspicious Activity Reports when required. Communicate with customers, meet strict KPIs, provide process feedback, and work flexibly across shifts.
Fintech • Mobile • Payments • Software • Financial Services
Recruiter responsible for end-to-end hiring for Wise’s Servicing Operations teams in Hyderabad. The role manages candidate experiences, partners with hiring managers and sourcers, builds talent networks, uses recruiting data and dashboards to optimize processes, and supports talent initiatives such as referrals, talent mapping, and salary benchmarking.
What you need to know about the Hyderabad Tech Scene
Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.
