Sr. Manager -- Big Data Application

TransUnion's Job Applicant Privacy Notice

Team Overview

An M02 is responsible for managing a development team, ensuring delivery of software solutions, and aligning execution with business and product goals.
The role balances:
* People management
* Technical oversight
* Delivery accountability
Typically owns a single team of engineers This is a hybrid position and involves regular performance of job responsibilities virtually as well as in-person at an assigned TU office location for a minimum of two days a week.

Role Overview And Core Responsibilities

• Lead a team of software and data engineers (typically 5–10 members).
• Ensure timely and high-quality delivery of platform capabilities, data processing solutions, and feature enhancements.
• Drive execution of large-scale distributed processing initiatives and data platform modernization efforts.
• Foster a culture of engineering excellence, innovation, and accountability.

• Provide technical leadership for large-scale distributed data processing systems built on Apache Spark.
• Guide architecture, design, and implementation decisions for scalable, fault-tolerant, and high-performance data processing workloads.
• Establish best practices for Spark application development, optimization, observability, and operational excellence.
• Ensure adherence to coding standards, performance requirements, scalability guidelines, and security controls.
• Lead troubleshooting and resolution of complex production issues involving Spark jobs, data pipelines, cluster performance, and resource management.
• Drive optimization of Spark workloads through partitioning strategies, query tuning, caching, memory management, and efficient resource utilization.

• Design and oversee development of batch and real-time data processing pipelines using Apache Spark.
• Collaborate on architecture involving Spark,  Iceberg, Hive, Hadoop,  AWS EMR, AWS Glue, GCP Dataproc, BigQuery, and other modern cloud-native data platform technologies.
• Ensure reliability, scalability, and operational efficiency of enterprise-scale data workflows.
• Drive adoption of data engineering best practices, CI/CD automation, testing frameworks, and monitoring for Spark workloads.
• Champion performance benchmarking, capacity planning, and cost optimization initiatives across data processing platforms.

• Work closely with product managers, architects, data scientists, platform teams, and business stakeholders.
• Translate business and data processing requirements into executable engineering plans.
• Communicate delivery status, technical risks, architectural decisions, and mitigation plans to leadership.
• Partner with cross-functional teams to deliver data-driven products and platform capabilities.

• Conduct performance reviews, coaching, and regular feedback sessions.
• Mentor engineers in distributed systems, Spark development, performance optimization, and software engineering best practices.
• Build technical depth within the team through knowledge sharing and career development.
• Support hiring, onboarding, and growth of engineering talent with expertise in big data and distributed computing.

• Ensure Agile, SDLC and engineering governance practices are followed.
• Track delivery, quality, reliability, and operational metrics.
• Improve engineering productivity through automation, standardization, and platform improvements.
• Drive continuous improvement initiatives focused on system reliability, performance, and customer satisfaction.

Required Knowledge And Experiences

• Strong hands-on experience with Apache Spark (Spark Core, Spark SQL, Structured Streaming, DataFrames, Dataset APIs).
• Proven experience building and operating large-scale distributed data processing systems.
• Strong understanding of Spark performance tuning, partitioning strategies, joins, shuffles, caching, memory management, and resource optimization.
• Experience with one or more programming languages such as Scala, Java, or Python.
• Experience with modern data ecosystems including Hadoop, Hive, Iceberg, AWS EMR, AWS Glue, GCP Dataproc, BigQuery, or equivalent technologies.
• Experience designing and maintaining batch and streaming data pipelines.
• Strong understanding of distributed systems, cloud-native architectures, and scalability principles.
• Hands-on experience deploying and managing Spark workloads on AWS and/or GCP.

• Proven experience leading engineering teams and delivering complex technical initiatives.
• Ability to balance hands-on technical leadership with people management responsibilities.
• Experience driving cross-functional collaboration and influencing technical direction.
• Demonstrated success managing technical roadmaps, execution planning, and delivery commitments.

• Mid-level management role positioned between Senior Engineer/Technical Lead and Senior Engineering Manager.
• Responsible for engineering execution excellence and technical leadership of Spark-based data processing platforms.
• Acts as a key decision-maker for delivery, architecture, operational reliability, and team growth.

• Experience with AWS services such as EMR, Glue, S3, Lambda, EKS, ECS, Step Functions, and CloudWatch.
• Experience with GCP services such as Dataproc, BigQuery, Cloud Storage, Dataflow, Pub/Sub, and Composer.
• Experience managing terabyte-to-petabyte scale data processing environments.
• Experience with Spark Structured Streaming, real-time analytics, and event-driven data processing.
• Contributions to Spark optimization, platform engineering, or open-source data ecosystem projects are a plus.

TransUnion Overview:

At TransUnion, we encourage and are committed to creating a real, positive impact and shared sense of purpose within our Workforce for Good, which empowers our people to grow, innovate and contribute to a better future for our communities and customers. We strive to build an environment where our associates are in the driver’s seat of their professional development— while having access to help along the way. We recognize that success comes when our associates thrive both professionally and personally; that’s why we prioritize work/life flexibility and offer resources for our teams across the globe to collaborate and drive excellence.

Be a part of our Workforce for Good – you’ll work with great people, pioneering products and cutting-edge technology.

TransUnion Job Title

Manager II, Applications Development