_0.png)
As a Senior Data Engineer, you will collaborate with product management, develop data pipelines, ensure code quality, and provide technical guidance while working in a DevOps environment.
About the RoleAs a Senior Data Engineer, you will work closely with engineering, analytics, data science staff across various Gap Development Center to deliver the goals and objectives of the business.What You'll Do
- Partner with Product Managers, and Solution Architects to define and analyze customer’s technical requirements
- Provides technical input to Statement of Work for External Service Providers
- Ability to break down product management stories into detailed design specifications for medium to complex business problems
- Experience with building data pipeline from several business applications
- Ability to collaborate with business stakeholders to identify and meet data requirements.
- Guides team members on the use of design patterns and adherence to code quality metrics
- Uses software code management tools and processes to ensure code traceability and good governance.
- Conducts code reviews with other team members (either internal or external service providers)
- Makes continuous improvement to our coding and test practices across applications
- Works in DevOps team which uses continuous development, integration and testing with automated releases to lower and higher environments with application monitoring plus supporting the product
- Experience performing root cause analysis on all data and processes to answer specific questions and identify opportunities for improvement.
- Participates in recommending emerging technologies and application solutions
Who You Are
- Candidate with 5+ years of experience in a Data Engineer role
- Graduate degree in Computer Science or equivalent
- High Problem-Solving Skills with analytical thinking and logical reasoning
- Having experience working on spark platform, spark sql, Python and common python libraries.
- Working experience on cloud-based environments such as Databricks on Azure. ADF, PySpark etc.
- Hands on experience writing notebooks using Python/Spark
- Well versed with relational databases, nonrelational databases, data streams, and file stores
- Experience with source control systems such as GitHub and Jenkins build and continuous integration tools.
- Automation using Linux shell & Python scripting.
Development experience with Enterprise integration patterns
Top Skills
Adf
Azure
Databricks
Git
Jenkins
Pyspark
Python
Spark
Spark Sql
Similar Jobs
Software
Design, develop, and maintain data pipelines and models while ensuring data quality. Collaborate with stakeholders for data needs and stay updated on data engineering practices.
Top Skills:
AWSAzureEltETLKafkaPower BIPythonSnowflakeSQLTableau
Marketing Tech • Sales • Software
Lead the development of scalable distributed data processing and architecture, while working closely with teams in an agile environment.
Top Skills:
AirflowAWSAzureGCPJavaKafkaPythonScalaSpark
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
The role involves developing the IOT Discover product, utilizing technologies for data ingestion, ensuring product quality, and collaborating with teams.
Top Skills:
AWSAzureCassandraDockerElastic SearchGCPGoJavaKafkaKubernetesPython
What you need to know about the Hyderabad Tech Scene
Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.
