Hands on: Siri Recaps & Live Rewind are controversial & unnecessary technologies

After spending a day with Apple's Audio Intelligence features, they feel more like checked boxes than fully realized and useful features in spite of the clear privacy and boundaries present.

When Apple first revealed Siri Recaps and Live Rewind during its September event, there was a lot of confusion and concern around the features. Now that they can be tested with the iOS 27.3 beta, I can say they are complicated to say the least.

Siri Recaps will listen while you're in a specific location, like work, or during a certain period of time you've set. Users can have it active all day if they wish.

Live Rewind requires the user to double-press the Digital Crown on Apple Watch to activate the feature. It displays the last 15 seconds of transcribed text, which can be saved.

These features require an Apple Watch Ultra 4 or Apple Watch Series 12. Now, "listens" comes with a ton of asterisks, because audio isn't being recorded or saved.

The secure exclave analyzes incoming audio, doesn't attribute speakers, and tosses the captured audio as it produces its summarized high points. Text transcripts of overheard conversations are also saved on a 15-second rolling basis, but are tossed on that buffer and can't be recovered.

Specific audio like sirens and alarms will trigger immediate notifications on Apple Watch, similar to alerts that HomePods can send out. The Audio Intelligence features will also note what music or podcast is heard when summarizing a conversation.

Okay, that's what these are and how they work, let's get into what using them is actually like.

Siri Recaps is AI at its most awkward

I went ahead and set up Siri Recaps to listen constantly regardless of my location or time. Don't worry, I've explained to the people in my life that I'm testing the feature.

That's the thing, it isn't "recording" audio, saving transcripts, or identifying speakers. Yet, somehow it feels a little like spying.

When I explained the feature to family, friends, and other colleagues, their first reaction was "I can see how I can make use of that." Then when I asked how they'd feel knowing another person could use it on them, they replied with "no thank you."

The strangest part of Siri Recaps is what it chooses to summarize and how it does so.

For example, a relative got into a stupid drunken fight and we were discussing what happened. It was a lengthy exchange followed by topics around finding cheaper housing for another family member, college classes, and a brief quip about a new Nintendo game launching soon.

The discussion of the altercation was mentioned in the recap, but barely as a blip. See the included image where it says "Drunken altercation details" followed by a single bullet point "keys, phones, and wallets were lost during the incident."

It didn't share where or why the fight occurred, the circumstances surrounding the event, or even that the individual did go back and find their items where they were left. I'm not sure if it is because it didn't hear those portions as clearly, or if the language used in the conversation made the summary throw out those portions.

Siri Recap won't include foul language, even censored, since it is just a summary. There's no reason to include it when it isn't quoting.

The details pulled from the rest of the conversation weren't helpful, but they did trigger some memory responses. Specific details like times classes start, costs of items, and other things were highlighted specifically.

Siri Recaps appears to be best suited for an office environment where conversations around finances and schedules are the norm. I could easily see the datapoints and information pulled from such an environment being useful.

It is not designed to work as a way to recall every conversation to 100% accuracy, especially casual ones. A rambling call I had with my mother entirely ignored a conversation about someone we knew's house burning down, but it did record what she said about needing to get her dog groomed.

Apple seems to have aggressively programmed the tool to avoid sensitive topics and details, though I'm sure it isn't foolproof. There have been plenty of conversations in the past day on important or sensitive matters, and they didn't show up, but there's no guarantee it won't mess up and capture something unwanted.

Also, since this is a summary, some details are conflated or incorrectly summarized. A detail about my niece moving high schools was tossed in as a consequence of a different niece moving to Virginia.

AI will always hallucinate when given the chance. It's why Apple has mostly avoided summarizing as a primary feature beyond notifications.

The good and bad of Siri Recaps

Technology isn't inherently good or bad, but its users can misuse or abuse it in plenty of ways. That said, Siri Recaps feels limited in how it might be abused.

Yes, an abusive spouse might use a summary of an argument to try and prove what was or wasn't said. Though, it's a summary, and quite the inefficient one at that.

Not that I'm saying "hey, go do this," but there are better tools available to aid in being a terrible spouse — many of which are on the App Store. Just like with AirTag, Apple entering this space didn't invent the idea of stalking.

I'm not trying to be dismissive of this issue. I have no doubt in my mind that some awful person will try to abuse Siri Recaps in this way, but I'm simply saying it's not going to be good for that.

On the other end of the spectrum is the usefulness of such a feature. If Apple is building what equates to constant personal surveillance tools, it should at least be useful or in demand.

I'm not sure that there is actually any real demand for such things. It all sounds good on paper, but in execution it leaves a lot of people open to harm.

The lack of consent in a constantly listening device isn't lost on me either. It just feels odd for Apple to have pursued this in the first place.

I've been wracking my brain for what might make such a feature worthwhile. The office notes example is a good one, though again, there are likely better tools.

Also, taking physical notes with a pen and paper will always make your brain retain the information more than handing off that cognitive load to an AI.

I do like the idea of using these summaries as a way to remember conversations and events for my Apple Journal. The iPhone already keeps a record of everywhere I go, everyone I text or call, and every website I visit.

However, those are all permissions I enabled on my personal device. There is no option for consent for the people also being summarized by AI in my life.

There's always an accessibility angle

I won't ignore the accessibility angle in the room. People with short-term memory issues or other cognitive problems might find such a tool very useful.

There's also a dark side to that too, as incorrectly summarized events or conversations could cause a lot of harm for some mental health conditions. It's a double-edged sword.

It's always a good idea to remember that what's good for some isn't good for others. Deciding to use this feature or not is very personal.

Live Rewind is even worse

Live Rewind is a parlor trick, but a creepy one. It is such an odd feature that even Apple failed to portray what it might be good for.

In my testing of Live Rewind, I can say this feature isn't ready for prime time. It often misunderstands certain words, especially homonyms.

Unlike Siri Recap, all audio is fair game. Double-pressing the digital crown after saying my social security number showed it clearly on the display.

Swear words are obfuscated with asterisks, though if you say a word that isn't spelled like the dictionary version of that swear, it'll pass through.

I tested the feature on Apple's own demo of the waitress at the restaurant. Here's the resulting text:

Hi, welcome. Today's specials are at the Eirline Tomato BLT, Caesar Salad, called Sell, lentil seed, chicken alfredo, the smoked salmon, and the avacado tartine."

I tried several times at various volumes and never got the correct list in one go. It's just not good for what it is.

Also, since you can save the transcript of text from Live Rewind, this can be easily abused. Sure, it plays an audible beep, but the deed is already done by the time the beep is heard.

I can see how easily Live Rewind can and will be abused. I cannot see when and where it might be useful, even when taking accessibility into account.

A controversial feature

I personally don't think Siri Recaps needs to exist. I'll continue testing it as it evolves through the beta and in the final release, but I'm not sure it'll be something I want in my life after that.

It will take a lot of heavy lifting from Apple and a positive reaction from society to make me reconsider. It just feels like too much to ask.

Meta Ray-Bans are called pervert glasses for a reason. I'd hate for Apple Watch to become known for secretly "recording" people.

Apple does seem to have an end-game here. There is a reason this technology exists beyond being another check box on the AI list.

Rumors suggest Apple will produce a set of smart glasses with cameras or sensors. The pitch would be the same — it lets AI "see" and summarize your day without saving video or photos.

If Apple goes that route, I expect it'll be nearly impossible to convince anyone they're different from the pervert glasses. Perhaps Audio Intelligence is a testbed for that sentiment.

The best thing we can do is use and test these features and let Apple know what we think. If the reaction is poor enough, it could be abandoned entirely.

I don't believe Apple needs to be in this particular business, but if this technology must exist, I suppose Apple would be the one I trust making it. A truly private and secure AI tool in a device I already own providing environmental information could be cool if it wasn't so damn creepy.

Time will tell how the world reacts.