By Robert Bruce – Chief Technology Officer
We’re thrilled to support Manchester Airport Group (MAG) in its journey of transformation. Collecting and analysing multiple data points across the passenger journey was the aim of this project. Using our Dev Ops approach, we ensured production workloads were running in AWS within 12 weeks of project inception!
About Manchester Airport Group
The Manchester Airports Group umbrella is the operations of Manchester Airport, alongside London Stansted and East Midlands. MAG Property and Cargo Operations come under this remit, representing Manchester Airport as a global gateway to the North of England. Manchester Airport has won prestigious industry recognition for customer service and holds the title of ‘Best UK Airport’.
The Challenge
MAG already had a mature stack of BI and database products for reporting. These were stretched and needed to be replaced by a solution that would create a scaleable and flexible data solution. This would allow MAG to reach its ambitions. The new solution would need to enable;
- an extended Data Warehouse
- scalable and elastic compute
- deal with seasonality spikes in passenger travel
This would allow real-time streaming of data; empowering MAG to become a real-time business across their entire customer journey.
This would give the control room and security staff a better view of the passenger flow and security performance. This allowed them to quickly make decisions based on real-time data. Data requirements included data ingestion from several on-premises systems and external data sources and then visualised on several large screens.
How we did this…the technical bit
Data Ingestion Challenges
After deciding on Elasticsearch as the data storage layer, we got to work on what the data ingestion needs where. The information was available from a variety of sources. On-premises database systems, files frequently put into AWS S3 Buckets, as well from external API data sources. An example is National Rail data, which we loaded into Elasticsearch using the STOMP (Streaming Text Oriented Messaging Protocol) interfaces.
This presented some initial challenges that we had to overcome:
- Polling for data from databases had to be more frequent than 1 minute
- Incoming data from STOMP was gzip compressed
Polling Databases More Frequently than Once per Minute
We addressed the first problem with a simple patch to the existing logstash-input-jdbc plugin. Before the patch, we could only express schedules in cron format. It used a Job Scheduler known as Rufus-scheduler, which supported supplying schedule expressions as either cron or number of seconds. The only change was a single line of code to use the repeat method instead of cron.
Handling Gzip Compressed Messages
To handle compressed messages arriving from STOMP interfaces, we had to decompress them to filter the data using Logstash. Although there is an existing codec for reading lines of data from a gzip compressed message, in our use case the gzip decompressed messages were multi-line XML. To overcome this, we created our own plugin — logstash-codec-gzip — which we’ve made available on GitHub.
Visualising the Data using Canvas
We started to use Kibana to visualise the data that was available in Elasticsearch hosted in Elastic Cloud. But we couldn’t quite do what we wanted. A more granular level of control was a requirement. We came across Canvas by chance, when the Head of BI from Manchester Airport Group and I were at the AWS Summit in London in May 2018. We spoke to the folks at the Elastic stand who mentioned Canvas and how they felt it was a better fit for what we wanted to achieve.
“We’re having difficulty in trying to visualise our data in Kibana as it doesn’t give the low-level control that we need”
“Have you seen Canvas?”
“No, what’s that?”
“Our new visualisation tool; it’s in technology preview at the moment, but it sounds like it might give you what you want”
Installing Canvas and First Impressions
Canvas was available as a plugin for Kibana and installed using the normal process of adding any Kibana plugin. Since Canvas was not available on Elastic Cloud during the technology preview phase, we hosted Kibana in the Manchester Airport Group AWS environment — fully scripted, using Terraform.
Canvas has a nice simple expression language for controlling how each element is visualised. It reminds me of shell programming where the output of one command is piped into the next, and sub-expressions are achieved by declaring an expression inside braces.
Our first attempt at a Canvas Dashboard for trains arrivals (see below for the current version)
After a couple of days of familiarising ourselves with Canvas, we were able to visualise the train arrivals at Stansted Airport. This dashboard was created using the Markdown element. This included the text, Image Repeat for the icons and images in the footer.
Things were looking great, and we had control of the individual elements, but there was still something missing. For our use case, we needed more control to provide:
- Format timestamps based on a time zone – data was being stored as UTC, but the time needed to adjust based on daylight saving
- Change the colour of the text and/or images based on known thresholds
Expanding Canvas
Writing plugins for Canvas is very similar to writing plugins for Kibana. They are written in NodeJS and utilise registries which allow you to add additional functions and even new elements that can be selected inside the Canvas UI. We created 3 plugins to achieve the level of control needed.
Using Time Zones when Formatting Timestamps
This was the first plugin we wrote, it’s a straightforward expansion of the built-in formatdate canvas function. It still uses moment.js behind the scenes, but also moment-timezone to allow the format of a timestamp in each time zone. Most of the plugin logic exists as follows:
import moment from 'moment';
import 'moment-timezone/builds/moment-timezone-with-data';
export const formatdatetz = () => ({
name: 'formatdatetz',
type: 'string',
help: 'Output a ms since epoch number as a formatted string according to a given timezone',
context: {
types: ['number'],
},
args: {
_: {
types: ['string'],
help: 'MomentJS Format with which to bucket (See https://momentjs.com/docs/#/displaying/)',
required: true
},
timezone: {
types: ['string'],
help: 'The timezone',
required: true,
default: 'UTC'
}
},
fn: (context, args) => {
if (!args._) return moment.utc(new Date(context)).tz(args.timezone).toISOString();
return moment.utc(new Date(context)).tz(args.timezone).format(args._);
},
});
The formatdatetz plugin is available in GitHub for you to install on top of Canvas.
Controlling the Colour of Text and Images
We couldn’t use Markdown to control the style of text and images (though you can provide cascade style sheets within canvas, Markdown syntax does not have support for setting HTML class attributes for which style to apply).
The simplest approach was for us to create a plugin that would allow direct coding of HTML and in a similar way to the Markdown element, allow handlebars expressions for data binding.
As SVG images can be embedded directly inside HTML, we could apply the same level of control to dynamically change the images based on data retrieved from Elasticsearch.
Creating the Visualisations
Once we had the plugins we wanted, having the control to create something simple as a table with different colours on the rows was now possible.
The first pass design for the security hub dashboard is based on user wireframes. The current dashboard for this is below, and it looks very different.
The visualisations were coming together. Data is populated into each of the relevant placeholders. But the look and feel still didn’t feel quite right and a request came in that reflected that:
“Can we make them look sexier?”
It was time to take inspiration from the internet, searching for a “modern dashboard” on platforms such as Pinterest and Dribbble gave some interesting approaches to designing a dashboard and a different direction was taken.
The security hub dashboard again, it’s now a dark theme, with the use of gauges and colour gradients (still not finished, it will change more)
Several elements were built-in to the screen, mostly using the HTML plugin created. A mixture of simple HTML and some complex SVG images dynamically generated.
The initial feedback was positive, however, users that hadn’t been part of the design didn’t understand all the metrics without an explanation.
“Wow! That looks a lot better, but what do the bottom of the bars mean?”
Going back to the original requirement of the Airport Security Operations being able to quickly make decisions, meant that although you could produce something that may look impressive, it’s not always the best way to visualise the information. Users instantly know what information the dashboard is displaying without it needing an explanation.
An Iterative Approach
Having become proficient in the Canvas expression language, we held a few workshops with live editing of dashboards with the stakeholders in the room. It was clear that we had moved on from the wireframe constructed before people had seen the tool or based on their current reporting, to inspired approaches on how best to display the information on the dashboards. By sharing dashboards with stakeholders and end users, we were able to collate feedback and make changes.
Final Dashboards for Manchester Aiport Group (MAG)
The following dashboards were the result of collaboration and creative thinking.
“The dashboards that synvert TCM (formerly Crimson Macaw) have created for us using Elasticsearch and Canvas have given Stansted Airport a modern real-time display of our data that was previously held in individual systems. This has created a number of efficiencies in our internal processes that would not have been possible before.”
– Stuart Hutson, Chief Technology Officer, Manchester Airport Group
Note, the data in all the dashboards is for presentational purposes only; using another Canvas plugin to do this!
Security Hub
This dashboard shows the status of Airport Security, from people entering and leaving the Airport Security to compliance rates, as well as information about the individual lanes. The shape chosen for the security lanes reflects the physical layout within the Airport Security so that the operating staff can immediately relate the dashboard to a physical lane.
Security Pod
This dashboard shows trend information, displaying people entering the Airport Security area plotted against a predicted forecast of passenger turn-up and queue wait times in different areas.
Flight Departures
Very similar to a normal FIDS (Flight Information Display Screen). This has the added detail of matching the percentage of passengers who entered Airport Security against predicted passengers for upcoming flights.
Train Arrivals
A large percentage of passengers arrive at Stansted Airport via train. Delays in their journeys have a significant effect on the number of people at Airport Security. A single train has hundreds of travellers and multiple trains arrive in a short time period after resolving issues.
The dashboard not only displays a typical table of train information but also shows train arrival times on a timeline. You can see this has changed substantially from the first iteration when we were experimenting with Canvas.
Technical Conclusion
Canvas is a brilliant tool to enable very impressive, real-time visualisation of data. Although Canvas is currently in beta, the ability to expand its functionality with plugins feels like the Elastic approach to software.
Lots of BI tools limit you to graphs and tables with a sprinkling of gauges and other standard visualisations. The simplicity of creating dashboards with Canvas allows Data Specialists freedom to create visualisations only limited by their imagination.
The Benefits for Manchester Aiport Group (MAG)
In the first 6 months of the project, MAG went from a single instance database to a scalable Data Warehouse. Daily sales rung in at store level to over 90% of all sales automatically ingested from over 50 separate retailers.
By the end of the first year, MAG had added a real-time streaming solution to ingest data from the car park and fast-track bookings. By the end of the second year, they had multiple real-time dashboards showing the state of the customer security queues. Data ingestion came from dozens of sources including trains, queues, metal detector scans, body scans, x-rays etc.
Stuart Hutson, MAG Chief Technology Officer, said:
synvert TCM (formerly Crimson Macaw) has accelerated our move to cloud for data warehousing with some great solutions, their work on real time dashboards using AI for passenger predictions in our airports has changed our operational agility and we look forward to doing more with them to make even better use of our data.
Want to know more? Need help with your own data journey? Get in touch here.
