Develop scalable batch and real-time data pipelines using Python and Azure data services. Integrate industrial systems such as SCADA, MES, Historians, and IIoT platforms with enterprise and cloud environments. Design data lake architectures, optimize performance, security, and scalability, apply data modeling expertise, troubleshoot complex issues, collaborate with stakeholders, and produce technical documentation.
ob Summary
RequirementsRequired Skills
Seeking an experienced Data Engineer with strong expertise in IT/OT Integration, Azure data services, and Python development. The role involves developing scalable data pipelines connecting industrial systems (SCADA, MES, Historians, IIoT) with enterprise and cloud platforms.
Key Responsibilities- Develop batch and real-time data pipelines using Python.
- Build solutions using:
- Azure Event Grid
- Azure Event Hub
- Azure Stream Analytics
- Azure Data Explorer (ADX)
- Azure Databricks
- Azure Data Lake Storage (ADLS Gen2)
- Optimize data processing, performance, security, and scalability.
- Collaborate with stakeholders and prepare technical documentation.
RequirementsRequired Skills
- Hands-on expertise in Azure data engineering services listed above.
- Advanced Python programming skills.
- Experience with real-time streaming and data lake architectures.
- Strong data modeling knowledge.
- Excellent analytical, troubleshooting, and problem-solving skills.
Preferred
- Manufacturing, Pharma, or Industrial Automation experience.
- Knowledge of SCADA, MES, Historian, IIoT platforms.
- Azure certifications preferred
Similar Jobs
Fintech • Information Technology • Insurance • Financial Services • Big Data Analytics
Designs and delivers enterprise-scale Azure lakehouse platforms, data products, ETL/ELT and streaming pipelines using Databricks, Spark, Delta Lake, and related Azure services. Leads and mentors data engineers, establishes engineering standards, drives Agile delivery, and collaborates with technical and business stakeholders. Implements CI/CD, security, governance, monitoring, data quality, lineage, observability, performance tuning, and cost optimization for production data platforms.
Top Skills:
SparkApplication InsightsAzure Data FactoryAzure Data Lake StorageAzure DatabricksAzure DevopsAzure MonitorAzure SqlAzure Synapse AnalyticsCosmos DbDelta LakeGitGithub ActionsLogic AppsAzureMicrosoft Entra IdPysparkPythonSonarqubeSpark SqlSQLSreUnity Catalog
Artificial Intelligence • Consulting
Design, build, and maintain Azure-based data infrastructure and pipelines. Implement data models and schemas, ensure data quality, optimize distributed storage and processing (Databricks/PySpark), and establish engineering best practices while partnering with cross-functional teams.
Top Skills:
AdfAdls2Ai FactoryAzureDatabricksDevOpsPygenticPysparkPythonUltimate
Artificial Intelligence • Consulting
The Senior Azure Data Engineer will design, develop, and maintain data infrastructure and pipelines, ensuring data quality, integrity, and alignment with business objectives. Responsibilities include collaborating with teams and establishing best practices for data engineering.
Top Skills:
AdfAdls2AzureDatabricksDevOpsPysparkPython
What you need to know about the Hyderabad Tech Scene
Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.
