Lead the design, development, and optimization of scalable GCP data pipelines, ETL/ELT workflows, BigQuery warehouses, and streaming solutions. Migrate on-premise systems to GCP, ensure data quality, security, governance, and reliability, and troubleshoot large-scale data platforms. Collaborate with technical and business stakeholders, establish data engineering best practices, and mentor junior engineers while guiding complex projects and recommending platform improvements.
Role Overview: We are seeking a highly experienced Senior Data Engineer with over 15 years of expertise in designing, building, and managing scalable data solutions. The ideal candidate will have extensive hands-on experience with Google Cloud Platform (GCP) services, particularly BigQuery, and a strong background in data pipelines, ETL/ELT processes, and data architecture.This role requires a strategic thinker who can lead complex data engineering projects, mentor junior team members, and collaborate with cross-functional teams to deliver high-quality data solutions.
Key Responsibilities:
- Design, develop, and maintain scalable and efficient data pipelines and ETL/ELT workflows using GCP services.
- Architect and implement data warehouse solutions using BigQuery, ensuring optimal performance and cost efficiency.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights.
- Optimize and manage large-scale datasets, ensuring data quality, integrity, and security.
- Develop and enforce best practices for data governance, data modeling, and data lifecycle management.
- Implement streaming data solutions using tools like Dataflow, Pub/Sub, and Apache Beam.
- Monitor and troubleshoot data pipelines, ensuring high availability and reliability.
- Lead the migration of on-premise data systems to GCP, ensuring seamless integration and minimal downtime.
- Mentor and guide junior engineers, fostering a culture of continuous learning and innovation.
- Stay updated with the latest advancements in GCP and data engineering technologies, and recommend improvements to existing systems.
Requirements
What we are looking for:
- 15+ years of experience in data engineering, with a focus on cloud-based solutions.
- Expertise in GCP services, including but not limited to:BigQuery, Dataflow, Pub/Sub, Cloud Storage, Cloud Composer (Airflow), Cloud Functions.
- Strong proficiency in SQL and experience with BigQuery SQL for complex queries and performance optimization.\
- Hands-on experience with Python or Java for building data pipelines and automation.
- Deep understanding of data modeling, data warehousing, and schema design.
- Experience with streaming data processing and tools like Apache Beam or Kafka.
- Familiarity with CI/CD pipelines and version control systems like Git.
- Strong knowledge of data security and compliance standards (e.g., GDPR, HIPAA).
- Proven experience in migrating on-premise data systems to GCP.
- Excellent problem-solving skills and the ability to work in a fast-paced, dynamic environment.
- Strong communication and collaboration skills, with the ability to work effectively with both technical and non-technical stakeholders.
Similar Jobs
.jpeg)
Artificial Intelligence • Hardware • Information Technology • Machine Learning
Own verification and end-to-end analysis of full-chip gate-level DRAM designs. Develop SystemVerilog and UVM testbenches, test cases, stimuli, test vectors, regressions, and coverage improvements. Simulate, debug, and analyze pre-silicon block and full-chip designs using digital, mixed-signal, and SPICE tools. Apply AI-based automation to accelerate verification while validating outputs against specifications, simulations, waveforms, assertions, and coverage. Guide newer engineers and collaborate globally on verification methodologies and environments.
Top Skills:
AmsCmosDramFinsimHspicePerlPliPythonReal Number ModelsSimvisionSpiceSramSystemverilogUvmVerilogVirtuosoVsimWaveviewXcelium
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Designs and operates scalable Databricks, Airflow, Snowflake, and Azure data platforms while developing AI/ML, Generative AI, RAG, API, and MLOps solutions. Responsibilities include building pipelines, vector databases, intelligent workflows, model deployment and monitoring, governance controls, and cloud-native AI services. The role collaborates with data, analytics, product, and platform teams, contributes to architecture and technical decisions, resolves production issues, and evaluates emerging AI technologies.
Top Skills:
Apache AirflowSparkAutogenAzure Ai ServicesAzure Api ManagementAzure App ServiceAzure FunctionsAzure Machine LearningAzure OpenaiCi/CdCrewaiDatabricksEmbeddingsGenerative AiGitInfrastructure As CodeLangchainLarge Language ModelsMlopsPrompt EngineeringPysparkPythonRagRest ApisSemantic KernelSnowflakeVector Databases
Artificial Intelligence • Consumer Web • Edtech • Enterprise Web • HR Tech • Social Impact • Generative AI
Lead enterprise content marketing initiatives across industries and buyer personas. Develop and manage data reports, e-books, videos, articles, webinars, and webpages; create briefs, messaging, and outlines; write and edit content; ensure brand consistency; and coordinate with product, regional, demand generation, customer marketing, sales, and creative teams. Use audience insights, SEO, AI-powered tools, and performance metrics to support full-funnel campaigns and editorial planning.
Top Skills:
Ai-Powered Content ToolsAioGeoSeo
What you need to know about the Hyderabad Tech Scene
Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.
