Build and manage data pipelines, enterprise connectors, raw-to-mart data layers, metadata and semantic indexes, vector databases, data quality controls, lineage, governance, and audit data stores. Support RAG and LLM applications with grounded, traceable data. Required skills include Python, SQL, modern data platforms, orchestration, API integration, testing, Git, and CI/CD. Bonus experience includes unstructured content processing, knowledge graphs, regulated life sciences data, and cloud or platform certifications.
Why this role matters
Every AI agent, scorecard, and dashboard we deliver is only as good as the data underneath it. As a Data Engineer, you’ll build that foundation: the pipelines, connectors, indexes, vector stores, and data marts that make our AI products accurate, grounded, and auditable.
What you’ll do
- Design and build ingestion pipelines for structured data, unstructured content such as documents and PDFs, and metadata
- Build reusable connectors to enterprise systems, catalogs, content repositories, and licensed third-party sources via APIs
- Model and build raw-to-mart data layers serving both analytics and AI use cases
- Implement metadata extraction, enrichment, and search indexing, including semantic and vector indexes for RAG
- Set up and manage the vector databases and knowledge repositories that power LLM agents
- Build data quality rules, profiling, scoring outputs, and exception handling
- Register lineage and maintain source registries, version tracking, and refresh controls
- Design data stores for signals, findings, audit trails, and user feedback loops
- Apply security and governance controls, including RBAC, PII and sensitivity flagging, and approved data handling
- Support SIT and UAT data validation, defect fixes, and production releases
What you bring
- Strong Python and SQL, with solid dimensional and normalized data modeling
- Hands-on experience with a modern data platform such as Databricks, Snowflake, or Azure/AWS data services
- Pipeline orchestration and transformation with Spark, Airflow, ADF, dbt, or similar
- REST API integration, JSON handling, and incremental or CDC ingestion patterns
- Data quality frameworks and pipeline testing practices
- Git and CI/CD for data workloads
Bonus points for
- RAG data preparation: chunking, embeddings, and vector databases (Azure AI Search, pgvector, Pinecone, or similar)
- Unstructured content processing, including text extraction, OCR, and document parsing
- Metadata management, data catalogs, ontologies, or knowledge graphs such as Neptune
- Experience supporting LLM or agentic applications with grounded, traceable data
- Life sciences data exposure and experience handling regulated data
- Azure Data Engineer, Databricks, AWS, or Snowflake certification
Similar Jobs
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Designs and develops large-scale ETL/ELT pipelines and data integration solutions using Azure Databricks, Spark, SQL, and cloud platforms. Builds data pipelines supporting machine learning, analytics, business intelligence, and data products; integrates diverse data sources and formats; optimizes data reliability, quality, and performance; develops AI-enabled automations; contributes to architecture and standards; and mentors other data engineers.
Top Skills:
SparkArtifactoryAzureAzure DatabricksAzure FunctionsAzure Service BusGitGithub ActionsGraphQLLakebaseLlmsPysparkPythonRedisRestSnowflakeSonarqubeSpark SqlSQLTerraform
Healthtech • Software
Leads the design and delivery of a cloud-native healthcare data platform. Responsibilities include building batch and near-real-time pipelines, implementing lakehouse architecture with Apache Iceberg, establishing data quality and observability frameworks, optimizing storage and query performance, and driving technical standards. The role also involves architecture reviews, code reviews, mentoring engineers, cross-functional collaboration, and information-security policy implementation.
Top Skills:
Amazon AthenaAmazon S3Amazon SagemakerApache AirflowApache HudiApache IcebergApache KafkaSparkAtlanAWSAws GlueAws Lake FormationAws MskAws Step FunctionsDbtDelta LakeIamPysparkPythonSQL
Artificial Intelligence • Cloud • HR Tech • Information Technology • Productivity • Software • Automation
Architects and builds scalable distributed data-platform systems, ingestion solutions, foundational libraries, and backend infrastructure. The role delivers high-performance Java services, operates Kubernetes-based platforms, applies fault-tolerance and high-availability strategies, and troubleshoots complex production issues. It also provides technical leadership, drives engineering best practices, evaluates emerging technologies, and integrates data lakes with analytics, query, and machine-learning platforms.
Top Skills:
Apache FlinkApache IcebergApache KafkaSparkAvroData LakesDevOpsJavaJvmKubernetesMySQLOracleOrcParquetPostgres
What you need to know about the Hyderabad Tech Scene
Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.
