Apple’s AI-driven “Audio Intelligence” technology for the Apple Watch has commenced beta testing, initially available for developers with the iOS 27.3 update, following its announcement at the company’s Surprise and Shine hardware event in September. This technology encompasses a suite of four optional AI features designed to assist Apple Watch Series 12 users in recognizing significant sounds, identifying music, catching up on conversations, and taking notes.
The sound recognition capability aims to aid users who are deaf or hard of hearing by enabling them to detect critical sounds such as sirens, alarms, or doorbells through their watch. Meanwhile, the song recognition feature utilizes Apple’s Shazam app to identify music playing in the background, providing details about the song title and artist.
However, the more contentious features involve listening to human conversations: Live Rewind and Siri Recaps. In light of the backlash against AI recording devices, such as Meta’s AI glasses and other wearables, Apple seeks to establish a balance where these features can be beneficial for Apple Watch users while maintaining the privacy of those nearby.
Live Rewind allows users to revisit recent dialogue, enabling them to catch what was said if they missed it. For example, if someone provides verbal directions or reads a lengthy list of specials at a restaurant, users can double-press the watch’s digital crown to view a text snippet of the last 15 seconds displayed on the watch face. Users have the option to either view this snippet, which will disappear permanently if not saved, or save it to the standalone Siri app introduced with iOS 27 for future reference, which could be useful for remembering names or jotting down quick notes during a fast-paced presentation.
Apple further states that users can select an “Ask Siri” option beneath the text snippet if they have inquiries about something mentioned—such as a technical term they do not comprehend or a sauce option referenced by a waiter. Upon double-pressing the digital crown, an audible chime and an animation will signal to those nearby that something has been recorded. If not saved, these snippets will automatically delete after 30 seconds.
The second feature, Siri Recaps, enhances this ambient listening technology by enabling Apple Watch users to capture high-level notes from their conversations. Importantly, Apple clarifies that it does not identify speakers, retain raw audio, or provide verbatim transcriptions. Instead, the feature summarizes conversations with a brief sentence and bullet points highlighting key information. For instance, bullet points may include significant dates, deadlines, pricing, or payment plan details, depending on the discussion topic. Apple suggests this feature could be beneficial in contexts such as work meetings, parent-teacher conferences, personal training sessions, or other situations where recalling tasks or statements is necessary.
Essentially, it aims to replicate a user’s ability to take notes without infusing emotion or tone. This could prove particularly advantageous when users are unable to write notes due to full hands or the need to focus on the conversation at hand. The recaps will not capture conversations while users are listening to music through headphones or utilizing noise cancellation features.
At the end of the day, users are prompted to review their saved Siri Recaps when the feature is activated, allowing them to choose which notes to retain. These notes can be stored in the Notes app or exported to the Journal app for more personal matters. If no action is taken, the recaps will automatically delete within seven days. Additionally, a reminder to review recorded Siri Recaps will appear when users first unlock their Apple Watch each morning.
Users have the option to manually enable Siri Recaps or set a schedule, such as limiting the feature to work hours or ensuring it does not operate at night. Even when scheduled, users can deactivate the recapping feature at any time via the Control Center. Apple emphasizes that Siri Recaps has been designed to exclude sensitive information, including government-issued identifiers, financial and authentication data, certain personal identifiers, and any language that could promote self-harm.
The Audio Intelligence features for the Apple Watch Series 12 and Apple Watch Ultra 4 rely on a combination of Apple’s hardware, software, and services, including the S11 chip, built-in microphone, and Apple Intelligence models operating on the iPhone, as well as Apple’s Private Cloud Compute.
The always-on processor contains a secure enclave—a hardware-protected memory space inaccessible to the operating system—where a dedicated audio buffer processes raw audio. This ensures that the audio is processed without being accessible to anyone, including Apple. The audio can only be accessed by the paired iPhone through a secure pairing mechanism between the watch and the phone. Lightweight AI models on the S11 chip detect if a conversation is occurring in which the wearer is involved, and this information is then transferred to the secure enclave on the iPhone. On-device processing on the iPhone converts the audio into condensed text, eliminating filler words. This condensed text is subsequently encrypted and sent to Apple’s Private Cloud Compute (PCC) to generate the conversation recap, which is then returned to the phone. (The recap and any summarizations used to create it are deleted from PCC immediately afterward.) The recaps accessible on the iPhone are exclusively available to the user and are encrypted during synchronization across multiple Apple devices.
Similarly, the Apple Watch transmits audio to the iPhone for processing to create the 15-second snippets for Live Rewind. Apple assures that none of these features will adversely affect the battery life of either the watch or the phone. The features will eventually be made available in a public beta for broader testing prior to their general release.
