Lead Data Engineer

Available in: UK - Home Based Worker, London, and Belfast

Apply by: 31 December 2017

The Lead Data Engineer designs and develops, as part of an agile team, substantial large-scale data processing components which meet user needs and quality standards. Is viewed as an authority figure for software development, providing strong development leadership for team members

Role Description:

  • Taking responsibility for development of whole components or software layers within a team. Development will incorporate design, coding, testing and defect fixing.
  • Focusing on hands-on design and development using open source and commercial platforms.
  • Being an authority figure for large-scale data processing and/or large-scale data storage.
  • Taking responsibility for the design and implementation of robust and performant ETL/ELT data processing pipelines.
  • Taking responsibility for data quality within the team.
  • Taking responsibility for unit and integration testing done within the team.
  • Taking responsibility for software product due diligence and integration.
  • Working with the Technical Architect to ensure data and software meets non-functional needs
  • Leads troubleshooting and tuning of large-scale data solutions to meet stringent non-functional requirements
  • Working with Ops to ensure the application software is operationally ready.
  • Works with InfoSec to ensure that any legal and security requirements have been applied against stored or processed data.
  • Advising customers and managers on the estimated effort and technical implications of user stories and user journeys.
  • Contributing to technical solutions as part of the sales process.     
  • Working with team members to advise on the estimated effort and technical implications of user stories and user journeys.

Required Experience:

  • Has software development experience with distributed data storage and processing technologies including: Hadoop, Spark, Key-Value stores.
  • Has software development experience in open source technologies including: JVM languages, Python, R, Ruby.
  • Has successfully developed substantial components for multiple large-scale data processing solutions.
  • Has been viewed and consulted as an authority figure for data stores and data processing development.
  • Has been viewed and consulted as an authority figure for NoSQL data stores.
  • Has data modelling experience with RDBMS and NoSQL data stores.

Essential Skills:

  • Has coached and mentored junior and experienced technical staff.
  • Have advised architects on technologies and development practices.
  • Able to make effective decisions within fast-moving Agile delivery.
  • Has good understanding of tools, design methodologies and best practice.
  • Able to simply and clearly communicate technical design in conversation, documentation and presentations.
  • Has an open attitude to sharing information.
  • Has respect of their team members and is able to provide technical leadership to a team.
  • Able to prioritise their own time and activities.

Desirable Skills:

  • Has software development experience with Cloudera’s distribution of Apache Hadoop.
  • Has experience of data visualisation and data wrangling.
  • Actively shares their thoughts and views on data best practices.
  • Has an understanding of general text engineering and associated NLP algorithms.
  • Has experience of security, performance tuning, capacity planning, resilience, and availability across multiple solutions.
  • Has participated in development and/or technology communities.
  • Has contributed to open source projects.

Other Information:

Our work is both challenging and rewarding. You’ll need a positive attitude and the ability to work independently, to adapt your style and practices in order to respond to changing circumstances.

You will need to be highly flexible, including willingness to travel frequently and work away from your home base.

Kainos is a professional services organisation with clients spread across the globe and we deliver projects both from client site, and from our offices. While we will attempt to base you on projects near or at your contracted office location, you need to be willing to travel to client sites and spend time away during the week if it is required.

Given the range and nature of work that we carry out for our clients, all Kainos employees are required to possesses up to date security clearance (Basic Disclosure, Access NI etc.), if you do not already possess this, you will be asked to apply for it prior to joining Kainos.

We’re an inclusive bunch at Kainos and we believe in equal opportunity. We welcome applications from all walks of life, including those with criminal records. Everyone who is offered a position here undergoes a background check, however no one is treated differently due to their background where it isn’t relevant to the role. It’s important to note – failure to reveal information which is relevant to the position sought, could result in us having to withdraw the conditional offer of employment.
In short: Having a criminal record will not exclude you from a career with Kainos. This depends entirely on the nature and requirements of the position, together with the circumstances and background of offences.