Apple will offer instant captions on iPhone, iPad and Mac

For several of its products, Apple announces a series of new accessibility features. Among these, Live Captions instant subtitles which are already known on Android smartphones (Android 10 or more) including Google Pixels.

On iPhone, iPad and Mac computer, the audio content may be entitled to a conversion into text with an overlay display of the subtitles for desired reading in near real time. Apple gives the example of a FaceTime call, video conferencing or social media application, for playing a streaming video.

Users will be able to adjust the size of the on-screen caption display. If such a feature responds primarily to a concern for accessibility, it may be relevant beyond for the automatic transcription of conversations in particular.

When live captions are used for Mac computer calls, Apple says users will be able to type a response with the keyboard and have it read aloud in real time to other participants in the conversation.

Apple’s Live Captions with Conditions

As with Android, Live Captions will be generated on the user’s device. There will therefore be no processing on Apple or other servers. However, it will take a iPhone 11 or higher, iPad models with A12 Bionic chip or later. For the Mac, it will be with Apple Silicon chip.

Initially, Apple’s Instant Captions will be available in beta and only in English. They will make their debut later this year.

Apple will also offer Door Detection for theiPhone Pro and iPad Pro with LiDAR sensor and always-on-device machine learning processing. The tool will locate doors in an unknown environment. With voice assistance, Door Detection will indicate whether a door is open or closed, how to open it and read any inscriptions on the door.

With Apple Watch Mirroring, it will also be the possibility to control the connected watch AppleWatch using support features on the iPhone (Voice Control, Switch Control…). Apple highlights an interest in access to health tools from the Apple Watch.