Loading…
This event has ended. View the official site or create your own event → Check it out
This event has ended. Create your own
View analytic
Thursday, March 10 • 14:50 - 15:30
Killing containers to make weather beautiful

Sign up or log in to save this to your schedule and see who's attending!

Abstract:
The Met Office Informatics Lab includes scientists, developers and designers. We build prototypes exploring new technologies to make environmental data useful. Here we describe a recent project to process multi-dimensional weather data to create a fully interactive 4D browser application. We used long-running containers to serve data and web pages and short-running processes to ingest and compress the data. Forecast data is issued every three hours so our data ingestion goes through regular and predictable bursts (i.e. perfect for autoscaling).

We built a Kubernetes cluster in an AWS group which auto-scales based on load. We used replication controllers to process the data. Every three hours ingestion jobs are added to a queue and the number of ingestion containers are set in proportion to the queue length. Each worker completes exactly one ingestion job from the queue and then exits, at which point Kubernetes creates a new one to process the next message. This has allowed us to remove the lifespan logic from the containers and keep them light, fast and massively scalable. We are now in the process of using this in our production systems.

Outline:
  • Introduce myself
  • Describe the Informatics Lab
  • Describe the problem (how to create a fully interactive browser application to explore 4D weather data)
  • Ingesting the data (supercomputer > binary files > pngs > videos > the browser)
  • Long container jobs (web servers)
  • Short container jobs (bin > png and png > video)
  • Optimising scalable hardware by mixing job types
  • Scaling Kubernetes based on message queues
  • Managing application lifecycle with Kubernetes
  • Persisting our data using Amazon Elastic Block Storage

Speakers
avatar for Jacob Tomlinson

Jacob Tomlinson

Senior System Engineer, Met Office Informatics Lab
The Met Office is a world leading weather forecasting and climate research organisation. It owns one of Europe’s largest supercomputers and generates data on the exabyte scale. Jacob Tomlinson is a developer and systems engineer at the Informatics Lab - the innovation unit within the Met Office. He leads their R&D efforts on new infrastructure technologies. Jacob uses Kubernetes and Docker to transform how the Met Office makes data accessible... Read More →


Thursday March 10, 2016 14:50 - 15:30
CodeNode - Cmd

Attendees (24)