Job Information
Nielsen Senior Spark Engineer - (Spark using Scala, Spark SQL and Presto) in Bangalore, India
You’ll be working within an international group of teams which span from India to Europe and US
As a Senior Spark/Lead Engineer, you will be working along and guiding a team of diverse engineers. This includes DevOps, Data, Backend and Front End engineers. Should be able to work independently and guide junior engineers. Should be passionate and driven to learn, suggest and adapt to new technologies
Responsibilities
Discuss the Cost of Change (= code quality) with your team members continuously
Write unit tests, integration tests and API tests
Should be able to support the application 24/7 based on team on call rotations.
Write clean code (mindful about coupling, separation of concerns, etc.)
You would be expected to write code 90% of the time, with an emphasis on Test Drive Development
Spend at least 10% of your time learning and improving existing application architecture
Should be open to learn and adapt to new technology architectures and patterns
High knowledge of Hadoop or any distributed architectures
Fair amount of experience with building and creating CI/CD pipelines
Should be able to conduct code reviews and design discussions
You would be expected to code and also analyze the impact of the change on data
You should be able to troubleshoot and solve complex problems in production.
You should be able to collaborate and coordinate and present ideas with different stakeholders including product, datascience, account managers etc
You should be able to diagnose AWS infrastructure issues related to the application
Follow Agile principles and feature related discussions and participate in grooming and planning sessions to effectively translate business requirements to Agile stories.
Follow best practices on 24/7 application monitoring and orchestration and performance optimization
Implement and Practice DevOps and SecOps for continuous incremental delivery and quality product with the guidance of senior engineers
Key Skills
Required
Bachelor or Master’s degree in computer science or related discipline (field), or equivalent work experience
4-8 years of Experience with Spark using Scala, Spark SQL and Presto. Upgrading, Maintaining and performance tuning large Spark/Scala applications is required.
4-8 years of advanced experience with Java and Relational databases is required.
4+ years of experience working with AWS services (RDS, S3) is required.
4+ Hands-on experience with Airflow DAGs and Python is required
2+ Experience using monitoring and alert orchestration tools such as Prometheus, Grafana, OpsGenie/PagerDuty tools is a must.
2+ Experience with building CICD pipelines in GitLab for applications running on Kubernetes (EKS) using Docker is required.
Ability to debug, performance tune large scale Java and BigData applications is needed.
Fluent in English, both spoken and written, with a large vocabulary (C1 english level)
Understand and implement basic object oriented principles and functional programming principles. Implements good coding practices with good unit and integration testing, with an emphasis on Test Driven Development.
Commitment to following best practices for security, scalability, and performance.
Excellent problem-solving skills and ability to troubleshoot complex technical issues in production environments.
Strong communication skills to collaborate effectively with cross-functional teams, stakeholders, and third-party vendors.
Continuous improvement mindset to identify opportunities for automation, optimization, and efficiency gains in infrastructure and deployment processes.
Ability to document processes, procedures, and technical architectures for knowledge sharing and future reference.
Preferred
Leadership qualities and the ability to inspire and motivate a team
At least 1 year of experience with Test-driven development
Familiarity working with large scale enterprise Java/BigData systems using Agile, TDD & DevOps methodologies.
Proven track record on delivering enterprise software solutions using Agile scrum methodology.
Nielsen
- Nielsen Jobs