A breakdown of Apple's Personal Knowledge Graph shows what your Mac and Apple Intelligence know about you and your life.
The upgrade to Siri AI added the understanding of personal context to Apple Intelligence. By pulling information from messages, emails, and photos, Siri can come up with answers to queries with contextual elements, such as knowing the name of your mother from a travel itinerary.
However, Apple has been compiling data on its users for a lot longer, with macOS 13 doing so since 2022. In a breakdown by security researchers at Eighth Rank, the generation of a Personal Knowledge Graph in macOS is detailed, including the kinds of data it compiles and retains on your Mac.
A pair of daemons
The process starts with a pair of daemons that Apple includes in macOS, used to compile the knowledge graph. However, Apple doesn't offer much information about them, aside from the name "knowledgeconstructiond" and a brief explanation that it analyzes device content and creates a knowledge graph.
The daemons store work in the Library/IntelligencePlatform directory, which contains 51 SQLite databases. While stored in plaintext, they are kept in an area protected by TCC Full Disk Access.
One the researchers found interesting was graph.db, using 181 entity classes and 278 predicates, with each row consisting of a subject, an object, and a predicate.
The subject is an entity of some form, such as a person or an app, with objects being a plain value like a name or a date. The predicate says the kind of relationship the database believes there are between the subject and the object.
A Personal Knowledge Graph visualization - Image Credit: Eighth Rank
The list of entities ranges from things like "person," "group of humans" and "organism" to topics such as "person in the arts," "sports organization," and "political party." Predicates include "what kind of thing this is," "genre," "has religion," "topics a person is interested in," and the "face cluster for a person."
The bulk of the work of the daemons is to populate the database with these relationships over time. Other databases include a communication log that pulls information on your location and movement, and a long list of device signals.
There's also a stream titled ContextualUnderstanding.PhysicalSocial, which appears to be a list of people and contact details that the user would've been physically near at some point. However, the researchers didn't see this filled in at all.
Families and deletion
The daemons would plainly copy data points from sources, and makes rulings based on what it finds. This includes inferring if a new data point is related to existing points, such as if a name is for someone already listed in the knowledge graph.
This can use a lot of different elements to produce a ruling. To determine who is family to the user, models apparently looked at 144 mostly behavioral features, as well as analysis from Photos of people who appear in pictures with you.
Each row is given a confidence score between 0 and 1.
Other models and systems are just as complex, bt there's also a bit of complexity to the way it handles deleted data.
Deleting some reference, such as a contact, doesn't immediately delete it from the database. The main live graph is capable of forgetting, with deleted contacts simply disappearing as they don't get derived again by the system for maintenance.
A second table, "expired_stable_graph," is kept at the same time as the live graph. It contains retracted rows, complete with confidence and relationship scores, with data seemingly pruned from it after 48 hours.
The result is that a conclusion derived from the models can still outlive the information on the device that helped produce it.
Detailed, but locally stored
The breakdown indicates that there's a massive trove of data stored on a Mac, and in surprising detail. In its research, it could determine there was a child in the household and its age, the daycare it went to, and associated phone numbers.
It was also capable of determining where a family would be, at what time, and for how long, as well as who would be at that place at that time.
From a privacy standpoint, it may seem concerning to have this amount of information and at such a granular level, all to drive things like Siri's personal context capabilities.
But at the same time, bear in mind this is locally-stored data on a Mac being analyzed here. It's a lot more private than what other AI-centric firms are producing in the cloud, because that data does end up leaving your device.
