Design, develop, optimize, and maintain scalable AWS data processing and ETL/ELT pipelines using Python, PySpark, Apache Spark, and SQL. Responsibilities include data extraction, transformation, validation, performance tuning, integration across databases, APIs, and cloud storage, monitoring, error handling, and data quality management. The role collaborates with engineering, architecture, analytics, and business teams while following Git, Agile, and software development best practices.
Job Title: Data Engineer
Experience Level: 4 - 6 years
Location: Hyderabad / Pune
Experience Level: 4 - 6 years
Location: Hyderabad / Pune
Qualification: BTech/MTech/MCA
Mode of Work: Hybrid
We are seeking an experienced AWS Data
Engineer with deep expertise in Banking, Financial Services, and Insurance systems. The candidate
must be familiar at least in one Data ware housing concepts
such as ODS/Marts and must have done at least one end-to-end
data processing pipeline designing.
He/she should be capable of handling the ETL design and
implementation solely and would work towards creating a
positive and innovation friendly environment. The incumbent
would be responsible for creating new capabilities.
Responsibilities
- Develop, maintain, and optimize data processing applications using Python and PySpark.
- Build scalable ETL/ELT pipelines to process large volumes of structured and unstructured data.
- Perform data extraction, cleansing, transformation, aggregation, and validation.
- Work with Apache Spark for distributed data processing and large-scale data transformations.
- Write complex SQL queries and optimize database operations.
- Develop reusable Python modules, functions, and libraries.
- Optimize Spark jobs by analyzing execution plans, partitioning, caching, and performance bottlenecks.
- Work with file formats such as CSV, JSON, Parquet.
- Integrate data from multiple sources, including databases, APIs, and cloud storage.
- Implement error handling, logging, monitoring, and data quality checks.
- Collaborate with data engineers, data architects, analysts, and business stakeholders.
- Follow coding standards, version control practices, and Agile development methodologies.
Requirements
Requirements
- Strong programming knowledge in Python, including functions, OOP, exception handling, and data structures.
- Hands-on experience with PySpark DataFrames, Spark SQL, transformations, and actions.
- Good understanding of Apache Spark architecture and distributed computing.
- Strong SQL skills, including joins, sub queries, CTEs, and window functions.
- Experience developing ETL/ELT pipelines and handling large datasets.
- Knowledge of data processing formats such as Parquet, JSON, and CSV.
- Experience with Git and software development best practices.
- Understanding of debugging, performance tuning, and troubleshooting.
- Experience with cloud platforms such as AWS, Azure, or Google Cloud.
- Familiarity with Databricks, Azure Data Factory, AWS Glue, or similar data engineering tools.
- Knowledge of data warehouses and lake house architectures.
- Experience with Airflow or other workflow orchestration tools.
- Familiarity with CI/CD pipelines, Docker, and automated testing.
- Knowledge of Delta Lake, Kafka, or streaming data processing.
Benefits
- Comprehensive Medical Coverage:Health insurance of INR 5.0 Lakhs for you and your family (up to 6 members), ensuring complete peace of mind.
- Robust Protection Plans:Group Personal Accident Insurance and Group Term Life Insurance to safeguard you and your loved ones.
- Retirement Benefits:PF and Gratuity provided as per standard government regulations.
- Flexible Work Options:Enjoy hybrid work arrangements & flexible working hours
- Generous Leave Policy:21 days of annual leave, in addition to 10 company-declared holidays.
- Employee Well-being Spaces:Access to a dedicated break-out area with round-the-clock refreshments for relaxation and rejuvenation.
DATAECONOMY Hyderabad, Telangana, IND Office
7th Floor, Smart Works, Aurbindo Galaxy, Sy No: 83/1, Knowledge City, TSIIC Raidurg,, Hyderabad, India, 500081
Similar Jobs
Insurance • Software
Build and maintain scalable AWS data platforms, pipelines, storage, orchestration, and infrastructure using Python, SQL, Terraform, Snowflake, Airflow, dbt, and related tools. Implement observability, data quality, governance, access controls, lineage, and cost optimization. Deploy containerized services, support streaming integrations, troubleshoot platform issues, and collaborate with senior engineers and cross-functional data teams.
Top Skills:
AirbyteAmazon AuroraAmazon EcsAmazon EksAmazon KinesisAmazon MwaaAmazon RdsAmazon S3Apache AirflowAWSAws IamAws LambdaClaudeCortex CodeDbtDbt CloudDockerGitGithub ActionsKafkaKubernetesPythonSnowflakeSQLTerraform
Information Technology • Consulting
Designs and maintains AWS data architectures, ETL/ELT pipelines, data warehouses, analytics datasets, dashboards, and visualization solutions. Uses SQL, Python, Spark, Glue, Lambda, S3, Athena, Redshift, and QuickSight to prepare, transform, validate, and govern data. Collaborates with product, engineering, and operations teams, optimizing performance, cost, scalability, and data quality while integrating data systems with AI and natural-language query tools.
Top Skills:
Amazon AthenaAmazon RedshiftAmazon S3Apache AirflowSparkAWSAws GlueAws LambdaAws QuicksightAws Step FunctionsPythonSQL
Information Technology
Lead development of an alarm-management ingestion, correlation, and rationalization engine for utility events. Define alarm schemas, asset and topology mappings, and sequence rules. Build low-latency processing for SCADA, ADMS, protection relay, and IED events using Kafka and Flink. Implement flood handling, deduplication, chattering detection, suppression, shelving, prioritization, and escalation. Tune performance for over 200 alarms per second, three-to-five-second event visibility, and operator guidance within 30 seconds.
Top Skills:
AdmsAmazon Api GatewayAmazon Aurora PostgresqlAmazon EventbridgeAmazon S3Amazon SnsApache FlinkAws GlueAws LambdaAws Step FunctionsConfluent KafkaIec 61968/61970 CimIec 62682Isa-18.2Kafka ConnectScadaSchema Registry
What you need to know about the Hyderabad Tech Scene
Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.
