Inside Visual Intelligence on iOS and macOS: Using Apple's best AI feature

Visual Intelligence is being promoted more thanks to iOS 27's changes, but it's also available for Mac users in macOS 27 Golden Gate. Here's what it is, and how it varies between the platforms.

Apple's WWDC keynote in June included a considerable amount of AI-related changes. The main change involved Siri AI, Apple's major upgrade to its digital assistant.

However, a second important element was also getting some much-needed changes. That would be Visual Intelligence.

Previously only available on iPhone, Visual Intelligence has been spruced up on that platform and brought to more. Users of macOS, iPadOS, and visionOS can also benefit from the image-based AI toolset.

This is everything you need to know about Visual Intelligence in the upcoming 27-generation operating systems.

What is Visual Intelligence

Visual Intelligence is part of a group of AI technologies that Apple has branded as Apple Intelligence. That group includes everything from its Foundation Models framework to Siri AI, from Shortcuts to App Intents, and from Image Playground to Genmoji.

Apple Intelligence, as a whole, is an entire system for AI processing, handling a multitude of tasks and queries for you. Visual Intelligence is an offshoot of it.

As the name implies, Visual Intelligence is all about using imagery to perform queries. While you may ask Siri a prompt in speech or text forms, Visual Intelligence also incorporates photographs, screenshots, and other images.

The key to remember is that Visual Intelligence is primarily about queries via images.

At its most basic, you can take a photo of something, and Visual Intelligence will do what it can to identify what's shown to it. Once it has identified a subject, it will then provide some information to you.

That said, you can still make text- or voice-based queries to add additional context to the image. This will help narrow down or nudge Apple Intelligence to give you specific information.

Broadly speaking, the kinds of interactions you have with Visual Intelligence are similar across devices. It's fundamentally the same system on different platforms, and you can get the same results.

Visual Intelligence in iOS 27

Starting with the release of the iPhone 16 generation, users were able to summon Visual Intelligence using the Camera Control button. Later, Apple expanded it to be available on the iPhone 15 Pro and iPhone 15 Pro Max, and also allowed it to be accessible using the Action button or Control Center.

Camera

For iOS 27, Apple has made it a lot easier to access the same functionality through a more obvious means. That is, directly in the iOS Camera app.

Using the mode selection element at the bottom, typically used to switch between Photo and Video, users can select Siri to go into an interface that is just for Visual Intelligence queries.

Tapping the Siri button will take a photograph, which Siri will then analyze for the subject and provide information based on what it can see.

If it's an object, it will try to identify it and give you some background information. For example, a bit of history about a deck of cards.

If it's an image with some actionable information, such as an address, it will do the same, but add in more options just above the main buttons. For example, the address could provide options to add to contacts or to open Maps to show that specific location.

In either case, the black Siri element at the top of the screen will have some information about the image, as part of a Siri Ai conversation. As normal for new Siri, you can swipe down to continue the conversation.

You can also tap the speech bubble button on the left to bring up Siri for a text query involving the just-taken image.

The right-hand button is an Internet search for the current shot. This will send the image off to Google and bring up listings relating to it under four tabs: Google, Amazon, eBay, and TikTok.

In any tab, select the image or link to bring up a browser or relevant app for more information.

Screenshots

While Visual Intelligence is chiefly about taking photographs of items in the moment, it's not just limited to shots of things. It also works in iOS itself.

Take a screenshot of the screen by pressing the right side button and the volume up button together. Then tap the small screenshot thumbnail in the corner.

You can tap Ask in the bottom left to ask Siri about the contents of the screenshot, with it opening the usual Siri AI dialog box.

If you tap Search in the bottom right, it will perform an Internet search, again with the four tabs as before.

Lastly, you can use your finger to draw a ring around specific things you want to search, if you only want part of an image or screenshot to be searched.

Photos

If you have previously taken a photograph or screenshot and saved it, you can still perform Visual Intelligence processing with it.

Open Photos, tap the image you want to query, and swipe up. Two buttons appear, titled Ask Siri and Image Search.

Ask Siri will bring up the Siri dialog box, like in previous sections. Likewise, Image Search brings up the Google, Amazon, eBay, and TikTok tabs.

Siri Query

Since a lot of Visual Intelligence is handled through Siri AI, it makes sense that the new Siri interface also works with it.

Within the Siri app, start a new query thread, or open an existing one. Tap the plus button next to the query entry text box.

Here, you can select Camera to take a photograph of something, which will be inserted into the Siri conversation. You can also select Photos to navigate your previous shots for the query, which again gets added to the conversation.

Visual Intelligence in iPadOS 27

The iPadOS 27 introduction of Visual Intelligence is, relatively speaking, similar to the iOS 27 experience. With a few exceptions.

First, you don't have an Action Button, Camera Control, or similar option to slip into the mode. There's also no Siri mode in the camera app either.

That said, the rest of the iOS functionality still works the same way. That is, you can access it when taking a screenshot, through the Photos app, or by inserting images into the Siri app itself using the Plus button.

You don't have the ability to take a photo and have Siri immediately scan it. But you can take a photo of a thing you want to know about and have Siri scan it afterward.

It's a mild annoyance, and hopefully something Apple will change in the future.

Visual Intelligence in macOS 27

While Visual Intelligence is chiefly known about on iOS, it's also something that Mac users can take advantage of. Albeit with a different way of interfacing with it.

Siri

The process of using the Siri app for Visual Intelligence queries is identical to that of using it in iOS and iPadOS. It's all about the small plus icon to the left of the text entry box and clicking it.

Doing so brings up options for Photos and for Files. On clicking Photos, you can then search through your Photos library for images to select, with the option to select multiple, before clicking Add.

After that, add any extra text to narrow down your query.

If you have a folder open on your desktop containing images, you can simply drag and drop them in. The images will automatically be added to the current query.

Photos

If you select a photograph with a right-click, you can bring up Ask Siri. This is something that's available elsewhere in macOS 27 too, but it's pertinent here since selecting the image makes it part of the query.

You can also select the Information option to show the usual metadata view of an image. At the bottom are buttons for Ask Siri and Image Search.

Ask Siri does the same as the right-clicking option. Image Search once again searches the Internet using the image.

Screenshots

Visual Intelligence is also accessible as part of the screenshot functionality of macOS. This means you don't need to have created an image beforehand, but instead can query based on whatever you can see on the screen.

That means you can screenshot part of a photo, a section of a document, or even an app interface, which you can then query.

While there's no camera-specific function to Visual Intelligence in macOS, there's nothing stopping you from doing something similar instead of photographing an object before the query. You could open Photo Booth and screenshot the preview to search for something visible by the Mac's webcam.

To use this, select Command-Shift-5 to bring up the screenshot menu. Select the right image-capture option (such as full app window or better, a defined part of the screen), and then select the Siri capture icon to the right.

The Ask Siri button should appear on the far right of the screenshot bar. Click it to start a Siri query.

To the side, a compact Ask Siri box will appear. Type your query in it, and it will answer as usual based on what it can see.

Depending on what is in the captured area, Siri AI may provide some initial suggestions of what you want to know about. Even if you don't start typing a query manually.

Usually, you will see the option for Image Search, which functions just like other instances of the command. However, it can also propose things like Nutrition if you capture an image of food, for example.

Select the proposed item to see what Siri AI has prepared for you.

More to come

This article covers the main ways you're going to encounter Visual Intelligence across Apple's main operating systems, but there's still more to uncover.

As time rolls on and Apple improves the capabilities of Visual Intelligence, you can expect more features to be added. You may be in for a long wait, as Apple will probably hold off on major updates until future major OS updates.

This article also doesn't cover the usage of Visual Intelligence in visionOS, used in the Apple Vision Pro.

The expansion of Visual Intelligence from the iPhone is a natural development. As Apple's AI work improves over time, so will Visual Intelligence's accuracy and capabilities.