As a Senior Data Engineer, you will monitor jobs, troubleshoot issues, support stakeholders, and develop solutions using AWS technologies.
Embark on an exciting journey into the realm of software development with 3Pillar! We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.
Job Summary
For this role, we are seeking a Senior Data Engineer for our Client's ETL Support Team to act as a technical pillar during our Client's transition to an AWS-native data environment. You will be a key escalation point, responsible for handling the most complex data pipeline issues. This role may move beyond simple support; you might be requested to harden and improve our Client's systems by driving performance tuning, cost optimization, and automation initiatives.
Remote Opening
Shift: Day Shift (9-hour shift, 5 days a week, weekends and holidays on rotation).
This role requires covering weekends and holidays on a rotational basis to ensure 7-day-a-week coverage.
The critical time window to be covered is 11:00 AM to 7:00 PM IST
Responsibilities:
- Job Monitoring: Monitor approximately 2,300 scheduled jobs (daily, weekly, monthly) to ensure timely and successful execution. Execute on-demand jobs as required by the business.
- Troubleshooting & RCA: Troubleshoot job failures, perform detailed root cause analysis (RCA), and provide clear documentation for all findings.
- Issue Resolution: Address and resolve bugs and data-related issues reported by the business team.
- Data Integrity: Verify source file placement in designated directories to maintain data integrity. Reload Change Data Capture (CDC) tables when structural changes occur in source systems.
- Data Synchronization: Help manage synchronization between external databases (including Teradata write-backs) and AWS Glue tables.
- Development: Assist in developing new solutions, enhancements, and bug fixes using AWS Glue, Lambda, and Step Functions.
- Stakeholder Support: Answer questions from the business and support User Acceptance Testing (UAT) inquiries.
- Decision Making: Make timely decisions to resolve issues, execute tasks efficiently, and escalate complex problems to senior or lead engineers as needed, all while maintaining agreed-upon SLAs.
- Documentation: Participate in knowledge transfer to internal resources and assist in creating necessary documentation and runbooks
Qualifications:
- Technical Skills: In-depth knowledge of AWS Glue, AWS Lambda, and AWS Step Functions.
- Core Concepts: A deep understanding of ETL processes and data warehouse design.
- Problem-Solving: Proven ability to troubleshoot data pipelines and perform root cause analysis (RCA).
- Experience: 3-5 years of relevant experience.
- Experience: Hands-on experience with Glue, Lambda, and Step Function development.
- Availability: Must be able to work a day shift that includes coverage for weekends and holidays on a rotational basis.
- Preferred Skills: Familiarity with Change Data Capture (CDC) processes, Teradata, and integration with SAS & RMJ events. AWS Certified.
Top Skills
Aws Glue
Aws Lambda
Aws Step Functions
Python
Teradata
Similar Jobs
Software
As a Senior Data Engineer, you'll automate data workflows, troubleshoot complex data pipeline issues, optimize performance, and mentor junior engineers.
Top Skills:
Aws GlueAws LambdaAws S3Aws Step FunctionsTeradata
Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Develop and implement advanced analytics, machine learning, and IIoT solutions in manufacturing, driving improvements and providing actionable insights for pharmaceutical production.
Top Skills:
Advanced AnalyticsAspenAws SagemakerDeltavIiotJavaScriptMachine LearningMatlabPlotlyPythonRRedshiftSnowflakeSpotfireSQLStreamlit
Fintech • Information Technology • Insurance • Financial Services • Big Data Analytics
Responsible for supporting data transmission and integration solutions, troubleshooting issues, collaborating with teams, and implementing new transmissions. Requires leadership and effective communication skills.
Top Skills:
AnsibleAzure Dev OpsData ExchangeElasticEnterprise Scheduling MaestroIbm Connect:DirectIbm SterlingIbm Workload SchedulerIisInformatica MftKubeLinuxOpen ShiftPowershellPythonSplunkSshSslTectiaWindows
What you need to know about the Hyderabad Tech Scene
Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.
