Senior Spark Architect
Location: City, US
Job Number: 28363
Position Title: Senior Development Architect
Everything Informatica does begins and ends with data. Simply stated, we make great data – data that is connected, clean and safe -- ready to use so that all enterprises can be data ready and put their unique information potential to work. A data ready enterprise is decision-ready, customer-ready, application-ready, cloud-ready and regulation-ready. And by design, our Intelligent Data Platform delivers great data to enable our customers to be ready for anything.
Cloud Data Integration Group is the innovation hub within Informatica that is responsible for delivering extensible, reliable, scalable, and innovative products. Our team specifically is responsible for Cloud Data Integration – Elastic (CDI-E), a cutting-edge cloud native data integration product built on top of the latest Spark on Kubernetes open source technology. Since our Engine heavily relies on open source code, especially Spark, your responsibility will be contributing to open source:
- Enhancing the Spark open source with new features and capabilities in order to improve our Spark Engine
- Developing Spark on Kubernetes open source code for optimal scaling and resource utilization
- Enhancing Apache Spark open source code to optimize jobs for best performance
- Enhancing Apache Spark open source code to reduce latency and processing time of Spark jobs to return instantaneous results
- Enhancing Apache Spark Open source to allow best execution and monitoring of Spark jobs and utilization of clusters for high performance and low TCO.
- Enhancing the Apache Spark Machine Learning capabilities
- Working with and enhancing latest open source technology to improve the product
Informatica Data Engine Platform has been a key constituent of Informatica’s platform and serves multiple products such as Data Integration, Data Quality, Data Masking, Profiling and MDM. Informatica products rely on the Spark Data Engine to fulfill the vision of executing data integration jobs across Clouds and on-prem in the most performant and cost effective way. You will be working with the latest open source technology on cutting edge innovative projects. You will be a key contributor to Informatica’s next generation Cloud platform and how it handles the volume, velocity and variety of structured and unstructured data. We are looking for you to look for ways to improve Apache Spark and other open source technology and contribute back to open source.
Our Ideal Candidate
You are an experienced software engineer who has made significant contributions to open source technologies especially in data processing, data management and distributed systems. You relish the opportunity to enhance and build on top of open source technology to add innovative functionality and optimizations that improve data processing performance across different use cases that include batch and streaming.
The successful candidate will be based in Redwood City, CA and will be responsible for:
- Contributing to key open source technologies
- Enhancing and building on top of open source technology to deliver innovative features for Spark Data Engine
- Using open source technology in incubating and designing the next generation cloud platform for complex heterogeneous systems and applications involving cutting edge technologies
- Collaborating with geographically dispersed, cross-functional teams in an Agile environment
- Mentoring junior engineers on technical, architectural, design and related issues and helping them contribute to open source
- Proactively identifying architectural weaknesses in open source technology we leverage and recommending appropriate solutions
- Providing architectural support and guidance for efficient deployments of Spark on cloud’
- Quickly learning, understanding, debugging and improving open source technology
- Taking ownership for open source component we leverage and enhancing them
- 10+ years of production experience in Java, Scala.
- Good understanding of the open source contribution process
- Deep knowledge and understanding of Apache Spark.
- 3+ years of hands on experience as a key contributor or committer with Apache Spark
- Ability to communicate and express thoughts and ideas in a big group
- Able to work independently with little direct supervision and take initiative; willing to mentor and develop others
- Strong analytical problem solving and decision-making skills
- Machine Learning knowledge is a plus
- BS in Computer Science or a related technical discipline; advanced degree preferred.
Alternative Location(s) :
Community / Marketing Title: Senior Spark Architect
Who We Are
Informatica empowers the world's most progressive companies to realize data-driven digital transformations that are changing the world. To do this, we live by our We "DATA" values. We Do Good, Act As One Team, Think Customer First, and Aspire For The Future. Together, we are conquering the impossible with data and changing what was once unimaginable into what’s now common—making lives richer, businesses stronger, and our world better.
Unleash Your Potential
A career with Informatica gives you all the opportunities and benefits that can only come from working for the trusted industry leader. By joining our team, you'll be able to solve real-life problems, make a difference, have a global impact, and join a supportive group of globally diverse teammates. We encourage you to be yourself, grow with us and unleash your potential.
EEO Employer Verbiage:
Navigating COVID-19 and Beyond
- Since March 2020, our INFA Team have been working remotely to do our part to slow the spread of COVID-19
- During this time, work-life balance and the well-being of our team has been a priority for us. In lieu of not being in the office, our teams are actively participating online via video chats. You'll find groups connecting for online games, virtual break rooms, online training, yoga, morning coffee, and so much more!
- We're also offering all teammates the ability to expense home office items (monitor, chair, desk, etc…) to ensure that you're as comfortable as possible
All qualified applicants will receive consideration for employment without regard to race, sex, color, religion, sexual orientation, gender identity, national origin, protected veteran status, or on the basis of disability.
Life at Informatica
Follow us to meet our team, learn more about life, careers, and events at Informatica. Conquering the Impossible with data, come join #LifeAtINFA!
Travel Requirement: Limited
Location_formattedLocationLong: City, US