This caused a complete pipeline shutdown within a matter of seconds of starting the data pipeline. However, in testing this library using the StreamSets Jython Evaluator, I found a certain feature used in the Java parsing library wasn’t thread-safe in Jython. I was lucky enough to find an open-source Java project to parse the UDP packet. While this worked, I was concerned about maintaining two separate pipelines (StreamSets and Python). First, I evaluated using Python (outside of StreamSets) to parse the F1 UDP packet.
#F1 2018 DEMO PC HOW TO#
The UDP packet is in a packed struct format with the spec provided by Codemasters, but there was still an open question remaining: how to parse these bits into usable data? When starting out this project, I didn’t know that it would work. Admittingly, having this first part of the data pipeline go from the UDP socket directly to Apache Kafka was a happy accident in the design, but ultimately ended up being a sound decision.We chose 60hz to make sure we had the fastest possible stream and richest data set to work with. The data stream has settings of 10hz/30hz/60hz (hz: units per second). Codemasters provides telemetry functionality within the game settings, so that users can add various hardware peripherals or dashboard apps by choosing an IP address/port to receive a stream of UDP packets. The first part of the data pipeline is getting the data out of the F1 2018 game from Codemasters.Listen on UDP socket, Write to Apache Kafka Producer.Additionally, I’ll discuss the engineering choices and tradeoffs we made while structuring the data pipeline.
The data pipeline for our F1 demo is comprised of three parts outlined below in a step-by-step guide. StreamSets allows anyone to connect a data Origin and Destination without necessarily having to write code. To accomplish the data engineering portion of the F1 demo, we used StreamSets, an amazing open-source data pipeline tool that I cannot say enough good things about. StreamSets: How to Build Amazing Data Pipelines (without serious DevOps chops!)
This powerful capability is made possible by the massive bandwidth and computing power provided by GPUs and harnessed by OmniSci.
#F1 2018 DEMO PC FULL#
You may have seen our blog post about VAST and vehicle telematics prior to NVIDIA GTC so I won’t recap the full post here, but our F1 demo was designed to demonstrate how OmniSci can ingest a stream of data AND allow that data to be analyzed interactively in real-time. Spatio-Temporal: data (often) has a Spatio- Temporal component which considers both time and 3D space.Agility: how to maintain the ability to interact with the data in an Agile manner.The goal of the F1 demo is to demonstrate a set of challenges that we refer to internally at OmniSci as VAST:
#F1 2018 DEMO PC CODE#
If you’re interested in the code details to replicate the demo yourselves, we’ve open-sourced the entire F1 demo on GitHub.
Each time we demonstrate the real-time streaming data capabilities of OmniSci via the F1 demo, we usually get one of two reactions: “You hacked a video game?” and “Hmm.better drive a lap to make sure this is real!” Since it’s difficult to explain the engineering behind the F1 demo over the roar of the engines and laughter as people drive questionably, here’s an outline of how to stream data into OmniSci in real-time using StreamSets.