Build and operate enterprise data pipelines using AWS Glue, PySpark, Delta Lake, Databricks, and Amazon S3. Responsibilities include transformations, schema enforcement, PII tokenization, deduplication, data merges, orchestration, governance, reconciliation, performance tuning, testing, production troubleshooting, CI/CD, deployment documentation, and Redshift loading.
Senior Data Engineer
AWS Glue / Databricks
LOCATION: Hyderabad, India
EXPERIENCE: 6–10 years (preferred)
ROLE OVERVIEW
Build and operationalize robust data engineering pipelines using AWS Glue, PySpark and Delta Lake to create governed,
high-quality enterprise datasets.
KEY RESPONSIBILITIES
- Develop AWS Glue / PySpark transformations from Raw to Refined datasets in Amazon S3.
- Implement schema enforcement, type conversion, PII tokenization, deduplication and merges.
- Develop Delta Lake pipelines and, when required, Gold datasets and Redshift loads.
- Orchestrate jobs using Airflow or AWS Step Functions with completeness and failure checks.
- Register datasets in Unity Catalog and support governance, metadata and consumer access.
- Perform reconciliation, performance tuning, unit testing and production troubleshooting.
- Prepare CI/CD artifacts, deployment configuration, runbooks and technical documentation.
Requirements
REQUIRED SKILLS
- Python, PySpark, Spark SQL and strong SQL
- AWS Glue, Amazon S3, Delta Lake and Databricks
- ETL/ELT, data modelling and data-quality validation
- Airflow or AWS Step Functions and failure recovery
- Git, CI/CD, data reconciliation and production support
PREFERRED EXPERIENCE
Unity Catalog, Amazon Redshift, Immuta, ServiceNow integration and regulated-data experience.
Similar Jobs
Fintech • Analytics
Senior individual contributor responsible for migrating and optimizing data pipelines onto Databricks and AWS lakehouse platforms. Designs batch, streaming, and event-driven ingestion patterns; implements governance, security, observability, testing, and CI/CD; integrates mastered and golden-copy data workflows; and applies AI-assisted engineering using LLM tools. Provides technical leadership, reusable engineering standards, production support practices, and guidance to distributed teams without direct people management.
Top Skills:
Amazon KinesisAmazon S3Apache IcebergAWSAws CodebuildAws CodepipelineAws GlueAws IamAws Lake FormationAws LambdaClaudeDatabricksDatabricks Unity CatalogDelta LakeGithub CopilotLlmsNeoxam DatahubPysparkSQL
Information Technology • Consulting
Develop and maintain enterprise data integrations and ETL/ELT pipelines connecting Salesforce CRM, CPQ, Billing, and other applications to centralized Azure Databricks platforms. Build solutions using Python, PySpark, SQL, Delta Lake, Lakehouse architecture, and Unity Catalog. Support data extraction, transformation, validation, governance, lineage, quality, mapping, and reusable data assets. Preferred experience includes Salesforce Revenue Cloud, finance and revenue datasets, canonical data models, data modernization, and consolidating duplicate pipelines.
Top Skills:
Azure DatabricksData GovernanceData LineageDelta LakeEltETLLakehouse ArchitectureMetadata ManagementPysparkPythonSalesforce ApisSalesforce BillingSalesforce CpqSalesforce CRMSalesforce ObjectsSalesforce Revenue CloudSOQLSQLUnity Catalog
Information Technology • Consulting
Build and optimize enterprise data engineering solutions using Oracle, Azure Databricks, SQL, PL/SQL, Python, and PySpark. Design ETL/ELT pipelines, complex transformations, and Silver and Gold data layers within a Delta Lakehouse architecture. Migrate Oracle business logic and data warehouse workloads to Databricks, implement governance through Unity Catalog, and perform schema analysis, validation, reconciliation, lineage, and performance optimization, including Finance datasets and reporting pipelines.
Top Skills:
Azure DatabricksDelta LakeEltETLLakehouseOracleOracle Autonomous Data WarehousePl/SqlPysparkPythonSQLUnity Catalog
What you need to know about the Hyderabad Tech Scene
Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.

