Discussion about Gesture Control with OpenVINO

[saadkhi]

Dear @openvino-dev-samples & Zhuo Wu,

I hope you are doing well. My name is Saad Ather Ali, and I am excited about the Gesture Control with OpenVINO project for Google Summer of Code. With my experience in computer vision and gesture-based interaction, I believe I can contribute effectively to this project.

I have been working on a project called Air Tracker ([GitHub Repository](https://github.com/saadkhi/AIR-TRACKER)), which is a hand gesture control system for project presentations. It enables users to interact with slides, multimedia, and on-screen elements using gesture recognition, eliminating the need for physical remotes or touch-based interfaces. My work on Air Tracker has given me hands-on experience with gesture-based navigation, media control, and custom gesture mapping, which aligns well with the objectives of this project.

Interest in This Project

The idea of porting MediaPipe Gesture Recognizer to OpenVINO and developing a local gesture control system for monitors greatly interests me. I see this as an opportunity to further optimize gesture recognition for real-time applications while enhancing the performance of gesture-based interactions using OpenVINO’s optimized inference capabilities.

Potential Contributions & Customization

With my prior experience, I would like to explore the following enhancements:

• Efficient Model Optimization: Improve the performance of gesture recognition using OpenVINO optimizations.
• Adaptive Gesture Control: Implement a dynamic mapping system that allows users to define their own gestures for various applications.
• Cross-Application Compatibility: Extend the system to control not only presentations but also media players, browsers, and accessibility tools.
• Multi-Device Integration: Investigate how this system can work across multiple screens or devices for a seamless user experience.

I have experience in Python, OpenCV, and TensorFlow, and I am eager to enhance my skills in C++ and OpenVINO while contributing to this project. I would love to hear your thoughts on additional areas I should focus on and any recommendations you might have for refining my approach.

Looking forward to your guidance and the opportunity to work on this project!

Best regards,
Saad Ather Ali

[saadkhi]

@zhuo-yoyowz and @openvino-dev-samples

[saadkhi]

@zhuo-yoyowz and @openvino-dev-samples

[zhuo-yoyowz]

Thank you for your interest in our project. Happy to see you have past experience on related projects.

[openvino-dev-samples]

hi @saadkhi sorry for late replay, I like your ideas.
Could you explain more on how to implement "allows users to define their own gestures for various applications." ? thanks.

[saadkhi]

Hi @openvino-dev-samples & @zhuo-yoyowz ,

Thank you for your response! I'm glad you liked my ideas.

Regarding "allowing users to define their own gestures for various applications", my approach would involve the following key components:

1. Gesture Recording & Customization

• Users can record new gestures by performing them in front of the camera.
• Each recorded gesture is mapped to a specific function (e.g., slide change, media play/pause, browser navigation).
• A UI tool (or a simple command-line interface initially) allows users to label and assign actions to gestures.

2. Model Training & Adaptation

• Instead of retraining a full deep-learning model, we can use feature extraction and a lightweight gesture classification layer that updates with user-defined gestures.
• OpenVINO can help in optimizing the updated model for real-time performance.

3. Gesture-to-Action Mapping

• A config file (JSON or YAML) will store user-defined gestures and their corresponding actions.
• The system reads this file to dynamically assign gestures without requiring a code update.

4. Application-Specific Integration

• The system can send gesture commands as keyboard shortcuts (e.g., spacebar for play/pause in media players).
• It can also expose an API or WebSocket for external applications to listen and react to gestures.

Would love to hear your thoughts! Should I focus more on a specific implementation aspect?

Best regards,
Saad Ather Ali

[saadkhi]

@zhuo-yoyowz can you tell me where i can find the discription of this issue (openvino good first issue) . i want to resolve this issue kindly tell.

[zhuo-yoyowz]

I think it's listed here on the issue webpage #28584
, about enabling operator for OpenVINO.

[saadkhi]

Dear @zhuo-yoyowz and @openvino-dev-samples ,

I’m eager to contribute to OpenVINO and strengthen my GSoC proposal, but I can’t locate the exact issue to solve. The initial comment references a "main issue," but it’s not linked. Could you clarify?

Also, I’d appreciate your feedback on my proposal:
Google Doc Proposal

Your guidance would mean a lot, looking forward to your insights!

[zhuo-yoyowz]

You could refer to this Good first issues (view), and find the submitted PRs "in Review" to see how relevant issues are solved for more info. Thx

[saadkhi]

@zhuo-yoyowz @openvino-dev-samples I have made the PR. kindly if you can review my proposal, your guidance would mean a lot. Proposal link : Google Doc Proposal