We are currently seeking a Data Engineer to join our data pipeline team whose main function is developing a real-time data processing and analytics system for the RealityMine market research data platform.
The ideal candidate will be working on complex problems in a challenging and fun environment, using some of the latest Big Data open source technologies like Apache Spark, as well as Amazon Web Service technologies including Elastic MapReduce and Redshift.
Key responsibilities include, but are not limited to:
- Developing software to receive, process and generate reports on data collected from panellists’ devices.
- Developing infrastructure automation and scheduling scripts.
- Supporting the production systems running the deployed software.
- Reviewing colleagues’ work and providing helpful feedback.
- Working with stakeholders to fully understand requirements.
- Interacting with team members to fully understand data structure.
- Adhering to company policies and procedures with respect to Security, Quality, and Health and Safety.
Required Skills and Experience
The ideal candidate has:
- Programming experience with data processing and manipulation.
- Knowledge of Spark, Scala* or OO language, such as Java or C#.
- SQL or Python development experience.
- Understanding of high quality coding and testing practices.
- Willingness to learn new technologies and methodologies.
- Good interpersonal skills, a positive attitude, and a willingness to help other members of the team.
* Scala is the main development language for our data pipeline, so if not already familiar, you will be expected to learn it as part of your work.
Preferable to have:
- Demonstrable experience in software development.
- Exposure to Apache Spark or another Hadoop-based data processing system.
- Knowledge of AWS or equivalent cloud technologies.
- Database technologies.
- Infrastructure automation.
- Agile software development practices, including continuous integration, automated testing and deployment.
Location: Manchester, England
Salary: Dependent on experience