Designs, builds, and optimizes scalable ETL and ELT pipelines using Python and PySpark. Develops high-performance applications for large-scale data processing, integrates AWS data services, writes advanced SQL, and builds REST APIs. Ensures cloud-based data solutions are secure, scalable, highly available, and compliant while collaborating with cross-functional teams to deliver business-focused data engineering solutions.
Python & PySpark Developer
Summary
We are seeking an experienced Python and PySpark Developer to join our IT Services team. In this pivotal role, you will architect and implement robust data solutions, focusing on the design, development, and optimization of complex data pipelines. You will play a critical part in transforming raw data into actionable insights by leveraging cloud-native technologies and advanced processing frameworks. This position requires a deep understanding of data engineering principles to ensure high availability, scalability, and efficiency across our data infrastructure.
Responsibilities
- Design, build, and maintain scalable ETL and ELT pipelines to facilitate seamless data movement and transformation.
- Develop high-performance data processing applications using Python and PySpark to handle large-scale datasets.
- Integrate and manage various AWS data services to support cloud-based data operations and storage requirements.
- Create and optimize SQL queries for data extraction, manipulation, and reporting within complex database environments.
- Implement and maintain REST APIs to enable efficient data integration between disparate systems and applications.
- Utilize cloud technologies to deploy and manage data solutions, ensuring security and compliance with industry standards.
- Collaborate with cross-functional teams to define data requirements and deliver data engineering solutions that meet business objectives.
Requirements
Requirements:
- Experience: 6 to 9 years of professional experience in data engineering or related software development roles.
- Core Programming: Proficiency in Python and Scala for developing data-intensive applications.
- Big Data Technologies: Strong expertise in PySpark for distributed data processing and analytics.
- Data Engineering: Demonstrated ability to design and implement ETL/ELT pipelines and data integration strategies.
- Cloud & Infrastructure: Hands-on experience with AWS data services and general cloud technologies.
- Database Skills: Advanced knowledge of SQL and relational database management systems.
- API Development: Competence in building and consuming REST APIs for data connectivity.
- Industry Context: Familiarity with the IT Services sector and its specific data handling challenges.
Similar Jobs
Edtech • HR Tech • Information Technology • Professional Services
Designs and optimizes scalable data pipelines using Python and PySpark, develops secure REST APIs, and deploys cloud applications on AWS. The role involves processing large datasets, managing distributed computing solutions, troubleshooting technical issues, optimizing performance, and collaborating with cross-functional teams to deliver reliable software and data engineering solutions.
Top Skills:
SparkAWSCloud InfrastructurePysparkPythonRest Apis
Fintech • Mobile • Payments • Software • Financial Services
Conduct customer and business onboarding through CDD and EDD reviews, assess identity documents, ownership structures, source of wealth, adverse media, and transaction activity. Maintain accurate KYC records, perform risk assessments, identify money laundering and fraud indicators, escalate risks, and file Suspicious Activity Reports when required. Communicate with customers, meet strict KPIs, provide process feedback, and work flexibly across shifts.
Fintech • Mobile • Payments • Software • Financial Services
Recruiter responsible for end-to-end hiring for Wise’s Servicing Operations teams in Hyderabad. The role manages candidate experiences, partners with hiring managers and sourcers, builds talent networks, uses recruiting data and dashboards to optimize processes, and supports talent initiatives such as referrals, talent mapping, and salary benchmarking.
What you need to know about the Hyderabad Tech Scene
Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.
