Design, develop, and maintain scalable batch and real-time data pipelines using Python and PySpark. Build cloud-based data processing solutions, CI/CD pipelines, and streaming systems while ensuring data quality, performance, and scalability. Collaborate with cross-functional teams to translate business requirements into reliable data architectures. The role involves technologies including Kafka, Flink, Kubernetes, Hadoop, Spark, MongoDB, and cloud platforms such as Azure, AWS, or GCP.
Job Summary:
Key Responsibilities:
We are looking for a skilled Big Data Engineer with strong experience in PySpark and data engineering. The ideal candidate will have hands-on experience in building scalable data pipelines, working with streaming data, and cloud platforms. Experience with Palantir Foundry is highly preferred
Experience Required: 5 – 10 years
- Design, develop, and maintain ETL/data pipelines using Python and PySpark
- Work with real-time/streaming data systems
- Build and optimize data processing solutions on cloud platforms
- Collaborate with cross-functional teams to understand business requirements
- Develop and maintain CI/CD pipelines
- Ensure data quality, performance, and scalability
- 5–10+ years of experience in Data Engineering
- Strong expertise in Python and PySpark
- Experience with Kafka, Flink, and Kubernetes
- Hands-on with Big Data technologies (Hadoop, Spark, MongoDB)
- Experience with cloud platforms (Azure, AWS, or GCP)
- Good understanding of data architecture and pipeline design
- Experience with Palantir Foundry
- Knowledge of Java
- Experience in consulting or Utilities domain
Similar Jobs
Information Technology • Professional Services • Consulting
Designs, develops, and maintains scalable data pipelines, data architectures, quality processes, and integrations across varied sources. Troubleshoots data issues, monitors solutions to meet SLAs, documents infrastructure, and collaborates with stakeholders and cross-functional teams. Requires strong SQL, database management, dimensional and relational modeling, and Python or Java experience, with preferred expertise in cloud data warehouses, big data, ETL tools, orchestration, BI dashboards, and Agile delivery.
Top Skills:
Agile ScrumAmazon RedshiftApache AirflowSparkAWSAws SnsAzure Data FactoryBigQueryClaude CodeConfluenceCortex CocoData LakeDatadogDbtErwinHadoopJavaJIRAJSONMatillionMySQLOraclePythonSnowflakeSQLSQL ServerSqldbmSsisTableauTeradataVisual Studio Code Copilot
Software
Design, implement, and lead large-scale generative AI and ML systems (LLM fine-tuning, RAG, agents, Graph ML). Build REST APIs, deploy with Docker/Kubernetes, optimize big-data workflows, mentor engineers, and operate ML services on cloud platforms.
Top Skills:
Anomaly DetectionAWSAzureBig DataCi/CdData MiningDockerDocument DatabasesFastapiGenerative AiGCPGraph MlKnowledge BasesKubernetesLlmsNoSQLPythonPyTorchRagRestful ApisSagemakerTensorrtTriton
Fintech • Financial Services
Support payroll processing, auditing, compliance, exception analysis, discrepancy resolution, employee inquiries, reporting, controls, and audit walkthroughs. Interpret wage and hour regulations, provide payroll subject matter expertise, guide less experienced staff, resolve complex issues, document procedures, retrieve audit artifacts, and recommend process improvements.
Top Skills:
Workday Time & Absence
What you need to know about the Hyderabad Tech Scene
Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.
