have Facebook announcement of A new machine learning project that aims to teach AI how to understand and interact with the world from a first-person perspective.
why it matters: Most computer vision is trained on images and videos taken from a third-person perspective, but in order to create AI assistants and robots that can work with us in the real world, researchers will need to compile data sets that can be used to create virtual machines. Known as egoistic perception.
What are you saying: Facebook AI’s Ego4D project involves a consortium of researchers from 13 universities and laboratories in nine countries working together to archive more than 2,200 hours of first-person video recorded in the real world with head-mounted cameras and other wearables. Used the company’s educational gifts for
- That data was complemented by an additional 400 hours of first-person video captured by Facebook Reality Labs Research using test subjects wearing augmented reality (AR) smart glasses in a staged environment.
- The resulting data is 20 times larger than any other existing Ego4D dataset in terms of hours of footage, and will be made publicly available in November to researchers who agree to Ego4D’s data use agreement.
What are they saying: “Down the line it will allow AI systems to help you in ways that are really in the moment, in terms of what you’re doing and what you’ve seen before, in all the ways you just can’t, ” says Kristen Grauman, Facebook AI research scientist.
Between the lines: As part of Ego4D, Facebook AI is establishing five benchmarks that researchers will be able to use to build and test egregious AI models using the project’s data sets.
- These include episodic memory (the ability to know what happened in the real world and when), predictability, hand-to-object interaction, audio-visual diurization, and social interaction.
- Put it all together, and you can eventually build AI assistants—perhaps tied to smart glasses like the ones Facebook and Ray-Ban recently put out—that can observe your actions and perform real-world tasks. may provide reminders or assistance.
- “Maybe you’re cooking a dish, and [the AI] It can anticipate that you’re going to end up needing key ingredients, and remember where they are physically,” Grauman says.
Bottom-line: It’s not hard to see how Facebook’s move into the AR and VR space will coincide with the creation of a first-person, egregious AI model.