On July 12, 2023, Arin Sime welcomed Paula Osés who is an AI engineer at Noumena to WebRTC Live, to explore how the data derived from computer vision is a game changer for industries ranging from telehealth to transportation. Also, how you can integrate computer vision algorithms with edge processing and WebRTC to enhance your video streaming capabilities.
Paula specializes in developing cutting-edge deep learning and computer vision algorithms. Her work at Noumena involves developing computer vision algorithms for different company projects, mainly focusing on object detection, segmentation and tracking. Additionally, she leads the Spanish team for the EU-funded project ICAERUS, which explores the use of drones in agriculture, forestry, and rural communities.
Paula spoke most recently at CommCon 2023. She shared her experience there (lots of AI!) and mentioned how interesting in particular Rob Pickering’s talk was on “Will GPT rescue machine voice interfaces?” (All of the CommCon talks are available on demand for a small fee.)
Watch Episode 81!
Computer Vision Project – Workflow
Paula shared the workflow for a computer vision project:
- Data Gathering
- Data Pre-Processing
- Image processing
- Video processing
- Subsampling
- Rescaling
- Encoding
- Algorithm
- Object Detection
- Semantic
- Pose Estimation
- Face Detection
- Subsampling
- Rescaling
- Encoding
- Data Post-Processing
- Clean data
- Extract meaning
- Create Visualization Segmentation
Use case: Urban mapping/traffic monitoring project in Barcelona
Paula discussed Noumena’s recent project analyzing spatial dynamics from video data recorded by the urban municipality of Barcelona, specifically the Department of Urban Development, Barcelona Regional. The scope of this research is to understand spatial dynamics in a public space, informing urban transformation and future planning decisions.
DATA SPECIFICATIONS
- Number of cameras: 7
- Camera sensor: RGB
- Time: 3 days (Monday, Thursday, Sunday) – 24 hours/day
- Streets: Super illa (Consell de cent), nearby streets (valencia)
- Amount of data: 600GB
Edge Computing will combat scalability and time of execution issues
Paula shared some of the limitations they encountered on the Barcelona traffic monitoring project, like scalability and time of execution.
As they start bigger projects, they are looking at edge computing as a way to improve time of execution, data privacy, and scalability. Edge computing processes data closer to where it’s being generated instead of a centralized location, enabling processing at greater speeds and volumes. Noumena is developing its own cameras with these edge processing capabilities built in.
The new workflow will look more like this:
Applying computer vision to real-time use cases
The remainder of the episode focused on how we can apply the data that can be collected via computer vision to real-time use cases. These include understanding the motion of the participants in a video, as well as facial recognition and emotion recognition which can be useful in video conferencing, webinars, classroom setting, exam proctoring, call centers, and much more. Who attended? Who is interested? When did the participants lose interest? Is someone cheating, or just coughing? How is a customer on a video call with a call center agent reacting to the solutions posed?
There are extra challenges in video as there are trade offs in needing to lower the latency to get more accuracy. Curiosity in the algorithms can also depend on the quality of the cameras. Multiple processes running makes the algorithm lag.
There are also privacy issues. Technology is capable of doing more than we want it to do from an ethical standpoint.
Bonus Content!
Midway through the episode, Arin debuted a new WebRTC.ventures ad entitled, “WebRTC Application Launch Day: A Cautionary Tale”.
The episode concluded with a discussion with WebRTC.ventures CTO Alberto Gonzalez about upcoming conferences where our team will be speaking, chairing, or attending.
Up Next! WebRTC Live Episode 82
Wednesday, August 23 at 12:30 pm Eastern
with guest Renan Dincer of Cloudfare.
