Multimodal Membership Inference Attack

Overall Research Idea

This research investigates if anonymized global positioning system (GPS) and inertial measurement unit (IMU) data is secure in an environment where video is being captured. In the event of a data breach for sports entities, can video footage from competitions be used to de-anonymize GPS/IMU data from an anonymous database? Prior literature has established that motion data can be fingerprinted in virtual reality games such as Beat Saber. Does this approach apply to data captured by sensors during sports competitions? And can this modality of data be fingerprinted based on video? We seek to find answers to these questions through our research project.

Current Work

Currently we have made progress on the following:

Developed an application for collecting GPS/IMU data analogous to that gathered during sports competitions.
Implemented metrics from Catapult to test the validity of movement collection compared to industry leaders.
Identified fingerprinting techniques to adopt to our specific use case.
Familiarized ourselves with OpenCap as a video tool to gather position data of body parts.

Highlight Video: OpenCap in Action

Preliminary exploration using OpenCap demonstrates the XYZ positions of joints, which will be crucial for proof-of-concept development.

Highlight Video: GPS/IMU Recording Application

Using an iPhone, this app provides data that matches that from Catapult and other GPS/IMU recording devices.

Action Plan

Our next steps include:

Reproduce fingerprinting techniques from work in virtual reality.
Create a pipeline to map OpenCap data to IMU data based on XYZ coordinates.
As part of an existing treadmill-based study, collect IMU and OpenCap data from participants.
Identify to what accuracy video can recreate IMU data.
Continue IMU data collection and translate VR fingerprints to the IMU dataset.
Create fingerprints from video and verify effectiveness against the IMU dataset.