Build and operate scalable batch and streaming data pipelines, connectors, and RAW-to-CUBE transformations using Python, Spark, Databricks, Kafka, dbt, SQL, and Delta Lake. Ensure data quality, schema validation, observability, performance, cost efficiency, and reliable recovery. Develop governed data products and feature-serving interfaces for machine learning models, agents, and downstream applications.
Job Summary
CONVO is seeking a Senior Data Engineer to build reliable data pipelines and connectors powering its CPG-focused agentic platform. You’ll develop scalable batch and streaming pipelines using Python, Spark, Databricks, Kafka, dbt, and Delta Lake, transforming source data into governed RAW and CUBE layers while ensuring data quality, performance, observability, and reliability. You’ll also build data-product and feature-serving interfaces that enable ML models, agents, and downstream applications to consume trusted data efficiently.
Technical mission- Build reliable connectors and RAW-to-CUBE pipelines across batch and streaming paths, and expose governed data and ML features to downstream services.
- Develop Python/Spark ingestion and transformation pipelines from source systems into RAW and Enriched CUBE layers.
- Implement streaming paths alongside scheduled workloads.
- Create dbt/SQL transformations, data-quality controls and contract-validation checks.
- Optimize Delta Lake layout, SQL performance, partitioning and incremental processing.
- Build feature-serving and data-product interfaces for models, agents and applications.
- Instrument pipelines for lineage, freshness, throughput, failure recovery and cost.
- Minimum experience: 5+ years in data engineering, including 3+ years delivering production Spark or python pipelines.
- Advanced Python and production Apache Spark/Databricks development.
- Kafka or comparable event-streaming technology.
- Strong SQL performance tuning and dimensional/data-product implementation.
- dbt and Delta Lake, including incremental patterns and table optimization.
- Batch and streaming reliability patterns: idempotency, checkpointing, replay and late-arriving data.
- Automated data quality, observability and schema-contract testing.
- ML feature stores or online/offline feature consistency.
- CPG, retail, ERP, POS or syndicated-data pipelines.
- Kubernetes-based data workloads and cloud cost optimization.
- Production connectors and RAW-to-CUBE pipelines with automated tests.
- Batch/streaming operational dashboards and recovery procedures.
- Documented data products and feature-serving interfaces.
- Performance and cost baselines for the implemented workloads.
- Works under the Data Architect with source-system owners, ML/optimization teams, platform infrastructure and downstream application teams.
Similar Jobs
Healthtech • Pharmaceutical
Senior Data Engineer responsible for making raw data usable and accessible across the organization. The role involves managing complex data engineering projects, contributing to policies and procedures, recommending improved processes and models, developing data acquisition, integration, and provisioning capabilities, and mentoring less experienced colleagues.
Food • Mobile
Design and build scalable data and AI infrastructure for production LLM applications. Responsibilities include developing RAG systems, embedding and vector retrieval pipelines, AI agents, LLM evaluation and observability frameworks, and batch or streaming data platforms using Databricks, Spark, Snowflake, Delta Lake, and Airflow. The role also involves optimizing distributed workloads, establishing data governance and quality practices, and partnering with engineering and product teams to productionize AI capabilities.
Top Skills:
Ai AgentsApache AirflowSparkCi/CdDatabricksDatabricks Mosaic AiDatabricks Vector SearchDelta LakeEmbeddingsGenerative AiJavaKafkaLangchainLanggraphLlamaindexLlmsMlflowPineconePythonQdrantRagScalaSemantic SearchSnowflakeSpark Structured StreamingSQLUnity CatalogVector DatabasesWeaviate
Software
Build, monitor, optimize, and maintain ETL/ELT pipelines using Google Cloud technologies. Develop complex SQL transformations, design dimensional data models, improve query performance and cloud cost efficiency, and ensure data integrity. Collaborate with analysts and stakeholders to translate SaaS and insurance metrics into technical solutions. Participate in code reviews, documentation, troubleshooting, and data warehouse lifecycle management while supporting US business hours.
Top Skills:
BigQueryCloud ComposerCloud StorageEltETLGCPGoogle Cloud DataflowIamSnowflake SchemaSQLStar Schema
What you need to know about the Hyderabad Tech Scene
Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.
