Build and maintain data foundations for AI agents, analytics, scorecards, and dashboards. Responsibilities include developing ingestion pipelines and enterprise connectors, modeling raw-to-mart layers, managing metadata and vector indexes, implementing data quality and lineage controls, and supporting governed data stores. The role also covers API integrations, incremental ingestion, security controls, testing, release activities, and data validation. Preferred experience includes RAG preparation, unstructured content processing, knowledge graphs, LLM applications, and regulated life sciences data.
Role summary: The Data Engineer builds the data foundation that AI agents, scorecards, and dashboards run on: ingestion pipelines, connectors, metadata and search indexes, vector stores, data marts, and quality instrumentation.
Experience: 3 to 6 years in data engineering. Senior Data Engineer: 2+ years, including ownership of data foundation architecture and connector frameworks.
Key responsibilities- Design and build ingestion pipelines for structured data, unstructured content (documents, PDFs), and metadata.
- Build reusable connectors to enterprise systems, catalogs, content repositories, and third-party or licensed sources via APIs.
- Model and build raw-to-mart data layers that serve analytics and AI use cases.
- Implement metadata extraction, enrichment, and search indexing, including semantic and vector indexes for RAG.
- Set up and manage vector databases and knowledge repositories used by LLM agents.
- Implement data quality rules, profiling, scoring outputs, and exception handling.
- Register lineage and maintain source registries, version tracking, and refresh controls.
- Design data stores for signals, findings, audit trails, and user feedback loops.
- Apply security and governance controls: RBAC, PII/sensitivity flagging, and approved data handling.
- Support SIT/UAT data validation, defect fixes, and production release activities.
- Strong Python and SQL; solid data modeling (dimensional and normalized).
- Hands-on experience with a modern data platform such as Databricks, Snowflake, or Azure/AWS data services.
- Pipeline orchestration and transformation (Spark, Airflow, ADF, dbt, or similar).
- API-based integration (REST), JSON handling, and incremental/CDC ingestion patterns.
- Data quality frameworks and testing practices for pipelines.
- Version control (Git) and CI/CD for data workloads.
- RAG data preparation: chunking, embeddings, vector databases (Azure AI Search, pgvector, Pinecone, or similar).
- Unstructured content processing: text extraction, OCR, document parsing.
- Metadata management, data catalogs, ontologies, or knowledge graphs (for example Neptune or other graph databases).
- Experience supporting LLM or agentic applications with grounded, traceable data.
- Life sciences data exposure (commercial, medical, regulatory, or launch data) and regulated-data handling.
- Cloud certification (Azure Data Engineer, Databricks, AWS, or Snowflake).
Similar Jobs
Big Data • Fintech • Information Technology • Insurance • Financial Services
Leads data engineering initiatives involving data architecture, modeling, integrity, and analytics platforms. Designs and implements automated ELT/ETL pipelines using Python, SQL, cloud platforms, workflow orchestration, and varied data stores. Performs SQL performance tuning, troubleshooting, and develops CI/CD workflows with Git. Independently drives complex data initiatives while applying strong analytical, problem-solving, communication, and multitasking skills.
Top Skills:
Amazon RedshiftApache AirflowAWSCi/CdEltETLGitNoSQLPythonSQLVertica
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Develop and maintain scalable enterprise data pipelines and ETL/ELT workflows using Python, PySpark, Databricks, Snowflake, Azure Data Factory, and related technologies. Support batch and streaming workloads, cloud data movement, platform stability, incident resolution, security remediation, upgrades, and operational support. Build AI tools, optimize data platforms, collaborate across teams, and contribute to documentation and continuous improvement within healthcare data environments.
Top Skills:
AirflowSparkAzcopyAzure Blob StorageAzure Data FactoryData ModelingData WarehousingDatabricksDelta Live TablesEltETLGitGithub CopilotIbm DatastagePysparkPythonSnowflakeSpark SqlTeradataUnix/Linux
.jpeg)
Artificial Intelligence • Hardware • Information Technology • Machine Learning
Develop applied data science, machine learning, generative AI, and agentic AI solutions for automated manufacturing environments. Responsibilities include data extraction and transformation, feature engineering, statistical modeling, anomaly detection, distributed processing, ML application development, and integration with Snowflake and Azure platforms. The role requires production-quality software, analytical problem-solving, and communication of technical findings to technical and business stakeholders.
Top Skills:
AdlsAi AgentsAngularSparkAPIsAzure Data FactoryAzure DatabricksAzure Machine LearningAzure OpenaiEtl/EltGenerative AiJavaScriptLlmsManufacturing Execution SystemsAzurePower BIPysparkPythonPyTorchRRagScikit-LearnSnowflakeSnowflake Cortex AiSnowparkSparklyrSparkrSQLSsisTableauTensorFlowTeradata
What you need to know about the Hyderabad Tech Scene
Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.
