The Data Engineer II will design and implement AWS-based data solutions, develop data pipelines, manage data lakes, and enforce data governance policies. Responsibilities include writing code in Python and SQL, optimizing cloud resources, collaborating with teams, and mentoring junior members.
Job Description:
The Data Engineer will lead the design and architecture of AWS-based data solutions, develop end-to-end data pipelines, manage data lakes, and optimize data platforms for performance and scalability. Responsibilities include writing and testing Python, SQL or other code, conducting code reviews, implementing end to end ETL/ELT processes, enforcing data governance policies, and driving innovation in data engineering practices. This role also involves collaboration with cross-functional teams, aligning with stakeholders on technical requirements, providing technical leadership, and mentoring team members to enhance overall team efficiency.
Key Responsibilities:
- Design and implement scalable data solutions on AWS, including data lakes, warehouses, and streaming systems.
- Develop, optimize, and maintain data pipelines using AWS services.
- Implement robust ETL/ELT processes and event-driven data ingestion.
- Establish and enforce data governance policies, ensuring data quality, security, and compliance.
- Optimize cloud resources for performance, availability, and cost-efficiency.
- Partner with cross-functional teams to gather requirements and deliver comprehensive cloud-based solutions.
- Identify opportunities to enhance systems, processes, and technologies while troubleshooting complex technical challenges.
Our current tech-stack:
- AWS: Glue, Lambda, Step Function, Batch, ECS, Quicksight, Machine Learning, Sagemaker, etc.
- DevOps: Cloudformation, Terraform, Git, CodeBuild
- Database: Redshift, PostgreSQL, DynamoDB, Athena
- Language: Bash, Python, SQL
Qualifications:
- Bachelor's degree in Computer Science, Engineering, or related field. Master's degree preferred.
- Expertise in AWS platforms, including data services. Basic knowledge of Azure is preferred.
- Extensive experience in data and cloud engineering roles.
- Expertise in AWS platforms, including data services.
- Strong competence in ETL processes, data warehousing, and big data technologies.
- Advanced skills in scripting, Python, SQL, and infrastructure automation tools.
- Familiarity with containerization (e.g., Docker) and orchestration (e.g., Kubernetes).
- Experience with data visualization tools (e.g., QuickSight) is a plus.
Top Skills
Athena
Aws Batch
Aws Ecs
Aws Glue
Aws Lambda
Aws Machine Learning
Aws Quicksight
Aws Sagemaker
Aws Step Function
Bash
CloudFormation
Codebuild
DynamoDB
Git
Postgres
Python
Redshift
SQL
Terraform
Similar Jobs
Artificial Intelligence • Consumer Web • Edtech • Enterprise Web • HR Tech • Social Impact • Generative AI
As a Data Platform Engineer, you will design and build a self-service data infrastructure for Udemy's data mesh. You'll create scalable data pipelines using AWS services and tools like Airflow and Kafka, and contribute to data quality and privacy initiatives. You will work collaboratively in a dynamic environment focused on innovation.
Top Skills:
AirflowAthenaAWSDatabricksDelta LakeEksEmrJavaKafkaPythonRdsRedshiftS3ScalaSnowflake
Consumer Web • eCommerce • Fashion • Retail
The Software Engineer II, Data Engineering role involves designing and maintaining data pipelines, integrating data sources for business insights, and collaborating with data teams to optimize systems for scalability and stability. The position requires strong software engineering skills, experience with big data technologies, and a focus on delivering quality real-time access to data.
Top Skills:
AirflowSparkAWSDatabricksDockerDruidEmrFlinkGoogle Apps ScriptHadoopKafkaKinesisRedshiftRubyScalaSQL
Fintech • Financial Services
The Lead Data Engineer will lead initiatives, develop tools for data and application problem discovery, review software enhancements, and collaborate with internal partners to solve data engineering challenges, requiring extensive software and database engineering experience.
Top Skills:
Apache KafkaAWSBig DataDatabase Management SystemsGCPHadoopMySQLNoSQLOracleSparkSpark SqlSQL ServerStreamingUnix
What you need to know about the Hyderabad Tech Scene
Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.
