Design, build, and optimize scalable ETL/ELT pipelines and distributed data platforms using Databricks, Apache Spark, Python, Java, and SQL. Develop ingestion, transformation, validation, monitoring, and data quality frameworks for large datasets. Tune Spark jobs and SQL queries, troubleshoot production issues, support cloud-based architectures, and collaborate with cross-functional teams. The role also involves mentoring junior engineers and contributing to engineering standards, governance, security, and operational excellence.
Job Role: Senior Data Engineer – Databricks
Location: Hyderabad, India (Hybrid)
Experience: 6+ Years
Number of Positions: 6
Role Overview
Anblicks is looking for experienced Senior Data Engineers to design, build, and optimize scalable data platforms and high-performance ETL/ELT pipelines using Databricks, Apache Spark, Python, Java, and SQL.
The ideal candidate will have strong hands-on experience in building enterprise-grade data solutions, processing large-scale datasets, optimizing distributed data workloads, and implementing reliable data ingestion and transformation frameworks. The role will involve working closely with data, cloud, analytics, and engineering teams to deliver scalable and high-quality data solutions.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and Apache Spark.
- Build robust data ingestion, transformation, processing, and validation frameworks for large-scale datasets.
- Develop and optimize data processing workloads using Python, Java, Spark, and SQL.
- Work extensively with Databricks to develop scalable data engineering solutions and production-grade pipelines.
- Implement efficient data transformation and storage solutions using modern data engineering practices.
- Write complex and optimized SQL queries for large-volume data processing and analytics.
- Perform performance tuning and optimization of Spark jobs, SQL queries, pipelines, and data processing workloads.
- Implement data quality checks, validation frameworks, monitoring, logging, and error-handling mechanisms.
- Work with stakeholders, data analysts, architects, and engineering teams to understand requirements and deliver effective data solutions.
- Contribute to the design and implementation of cloud-based data platforms and modern data architectures.
- Troubleshoot production issues and ensure the reliability, scalability, and performance of data pipelines.
- Follow data engineering best practices related to security, governance, data quality, and operational excellence.
- Mentor junior engineers and contribute to technical standards, reusable frameworks, and engineering best practices.
Required Qualifications
- 6 - 10 years of professional experience in software development and/or data engineering, with proven experience delivering production-grade data solutions.
- Strong hands-on experience in Data Engineering and ETL/ELT development.
- Strong experience with Databricks and Apache Spark.
- Strong programming skills in Python and Java.
- Advanced proficiency in SQL, including query optimization and working with large datasets.
- Experience designing and developing scalable data pipelines and distributed data processing solutions.
- Strong understanding of data warehousing, data modeling, and large-scale data processing.
- Hands-on experience with data ingestion, transformation, data quality, and pipeline optimization.
- Strong troubleshooting, debugging, and performance tuning capabilities.
- Experience working in Agile development environments and collaborating with cross-functional teams.
Preferred Qualifications
- Experience working with AWS cloud services and cloud-based data platforms.
- Experience with Delta Lake and modern Databricks data engineering capabilities.
- Knowledge of Databricks Workflows, Jobs, notebooks, clusters, and deployment practices.
- Experience with CI/CD, Git, and DevOps practices for data engineering.
- Exposure to AdTech, MarTech, Customer Data Platforms (CDP), or customer analytics environments.
- Experience working with high-volume, complex, and diverse datasets.
Education
Bachelor’s degree in Computer Science, Computer Engineering, Software Engineering, or a related technical field, or equivalent professional experience.
Anblicks Hyderabad, Telangana, IND Office
Raidurg Panmakhta, Serilingampally Mandal, Ranga Reddy, , Hyderabad, Telangana , India, 500081
Anblicks Hyderabad, Telangana, IND Office
Hyderabad, Hyderabad, India
Similar Jobs
Fintech • Mobile • Payments • Software • Financial Services
Conduct customer and business onboarding through CDD and EDD reviews, assess identity documents, ownership structures, source of wealth, adverse media, and transaction activity. Maintain accurate KYC records, perform risk assessments, identify money laundering and fraud indicators, escalate risks, and file Suspicious Activity Reports when required. Communicate with customers, meet strict KPIs, provide process feedback, and work flexibly across shifts.
Fintech • Mobile • Payments • Software • Financial Services
Recruiter responsible for end-to-end hiring for Wise’s Servicing Operations teams in Hyderabad. The role manages candidate experiences, partners with hiring managers and sourcers, builds talent networks, uses recruiting data and dashboards to optimize processes, and supports talent initiatives such as referrals, talent mapping, and salary benchmarking.
.jpeg)
Artificial Intelligence • Hardware • Information Technology • Machine Learning
Build and architect enterprise-scale agentic AI and multi-agent systems using Claude, MCP, AWS AgentCore, RAG, and GraphRAG. Design hybrid-cloud architectures across AWS, GCP, Kubernetes, and on-premises platforms; develop engineering-data ETL pipelines; and lead PoCs, technology standards, cross-functional collaboration, and mentoring across engineering, manufacturing, product, validation, and data teams.
Top Skills:
A2AAgentic Ai FrameworksAmazon NeptuneAWSAws AgentcoreClaudeClaude CodeETLGCPGenerative AiGraphragKubernetesMcpNeo4JOn-Premises PlatformsOntologiesPythonSemantic ModelingTaxonomiesVector DatabasesVector Rag
What you need to know about the Hyderabad Tech Scene
Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.
