Apple created a seismic wave of excitement with its groundbreaking announcement of the Vision Pro, a remarkable spatial computer. As the world marveled at the device’s technical specifications and its elegant engineering, a significant aspect of its design went relatively unnoticed: the innovative control interface. During the product demonstration, users were astounded as they effortlessly controlled the device using hand gestures, thanks to the presence of depth sensors cleverly integrated at the bottom of the device. However, what truly set the Vision Pro apart was its primary mode of interaction—through the eyes.
Unveiling the internal camera array artfully positioned within the headset, Apple revealed that the Vision Pro could accurately discern user intent by tracking the subtle movements of their eyes. This breakthrough eye-tracking technology represented a remarkable leap forward in human-computer interaction. Yet, Apple’s ingenuity did not stop there. Behind the scenes, the company harnessed the power of advanced machine learning algorithms to refine and perfect the eye-tracking functionality. By leveraging these algorithms, Apple ensured consistent and precise performance, pushing the boundaries of what was thought possible.
Moreover, the implications of Apple’s machine learning-powered eye-tracking system extend far beyond the Vision Pro itself. This groundbreaking technology has the potential to lay the foundation for future advancements in brain-computer interfaces (BCI). With its ability to accurately interpret and translate the intricate movements of the user’s eyes into actionable commands, the Vision Pro sets the stage for a future where BCI becomes a tangible reality. By seamlessly integrating cutting-edge machine learning algorithms with eye-tracking capabilities, Apple has opened the door to a world of possibilities, propelling the Vision Pro into the realm of a proto-BCI.
As the world eagerly awaits the arrival of the Vision Pro and its eye-tracking marvel, it becomes clear that Apple’s vision reaches beyond the confines of current technology. By seamlessly merging innovation, user experience, and machine learning, Apple continues to push the boundaries of what is imaginable. The Vision Pro serves as a testament to their relentless pursuit of groundbreaking advancements, offering a glimpse into a future where human-computer interaction transcends traditional boundaries, paving the way for remarkable new possibilities.
There are 5000 patents for a reason
Brain-computer interfaces (BCIs) have long been the stuff of science fiction, with projects like Elon Musk’s Neuralink capturing significant attention. Neuralink aims to achieve BCI functionality through the surgical implantation of a chip in the human brain, allowing thoughts to be translated into software commands. However, Apple appears to have made strides in non-invasively detecting human thoughts by combining neurotechnological solutions with machine learning algorithms.
Examining patents filed by Apple for the Vision Pro headset, one patent, in particular, stands out—’eye-gaze based biofeedback.’ Filed in Europe, this patent seeks to determine a user’s attentive state while viewing specific types of content. Apple states that this biofeedback can be used to track user responses in extended reality (XR) experiences, ultimately enhancing the richness of those experiences.
One example provided in the patent demonstrates the ability to predict user interaction based on the dilation of their pupils. Additionally, the patent suggests that the colour of the user interface (UI) element could change based on which colour triggers a more pronounced pupillary response, thereby increasing the system’s success rate. Similarly, patents have been filed for a system capable of assessing a user’s state by sensing how changes in lighting affect their pupils. This allows the system to determine if the user is inattentive and, if so, increase the luminance of a specific UI element or part of the content to regain their attention.
Pupil tracking is just one of the methods Apple researchers have identified to determine a user’s mental state. Another filed patent, titled ‘sound-based attentive state assessment,’ describes a system that assesses a user’s response to sound in order to gauge their mental state. The sensors can also measure other factors such as heart rate, muscle activity, blood pressure, and electrical brain activity to gather more comprehensive information.
From the collective information provided by these patents, we can begin to piece together the potential user interface of the Vision Pro. These biofeedback assessment systems could work in tandem, feeding information about the user’s mental state back to the headset. This comprehensive understanding allows the computer to adjust the content accordingly, providing a more personalized and tailored experience based on the user’s thoughts and feelings.
Sterling Crispin, a software engineer specializing in neurotechnology for the Apple Vision Pro, is the lead inventor listed on these patents. In a tweet, he provided further insight into this interface, mentioning additional techniques to infer cognitive state, such as quickly flashing visuals or sounds to the user and measuring their reactions. He described it as a “crude brain-computer interface via the eyes, but very cool.”
The way forward for BCI?
As Sterling Crispin pointed out, the non-invasive approach taken by Apple’s Vision Pro is a preferred method of interfacing with computers compared to invasive brain surgery, as exemplified by the challenges faced by Musk’s Neuralink. The difficulties encountered by Neuralink, including ethical concerns regarding animal testing and the hurdles in reaching human trials, highlight the extensive work still required to achieve a fully functional brain-computer interface (BCI).
Apple’s solution with the Vision Pro not only represents their distinctive approach but also paves the way for the future of human-computer interaction. Artificial intelligence (AI) and machine learning (ML) play a pivotal role in bridging the gap between organic brains and inorganic processors. This can be observed in advancements such as voice inputs, where what was once a computationally demanding task now becomes feasible even on edge devices like mobile phones, thanks to on-device processing capabilities. Lightweight speech-to-text (STT) algorithms have played a crucial role in this progress, continuously improving over time.
By combining sophisticated sensors, scanners, and machine learning algorithms, the Vision Pro is capable of interpreting the mental states of users, effectively qualifying it as a brain-computer interface. Similar to how STT algorithms revolutionized voice interfaces, Crispin’s research and Apple’s advancements hold the potential to revolutionize visual interfaces as well. Furthermore, this technology may very well become the primary method of interacting with computers in the future, surpassing traditional input devices like keyboards and mice, and ushering in a new era of intuitive and immersive human-computer interaction.
