搜索与研究
visual-intelligence
Integrate your app with iOS Visual Intelligence for camera-based search and object recognition. Use when adding visual search capabilities.
rshankras · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add rshankras/claude-code-apple-skills --skill visual-intelligence技能说明
name: visual-intelligence description: Integrate your app with iOS Visual Intelligence for camera-based search and object recognition. Use when adding visual search capabilities. allowed-tools: [Read, Write, Edit, Glob, Grep, Bash, AskUserQuestion] last_verified: 2026-07-16 review_by: 2027-06-22
Visual Intelligence
Integrate your app with iOS Visual Intelligence to let users find app content by pointing their camera at objects.
When This Skill Activates
- User wants camera-based search in their app
- User asks about visual search integration
- User wants to surface app content in system searches
- User needs to handle visual intelligence queries
Overview
Visual Intelligence lets users:
- Point camera at objects or use screenshots
- System identifies what they're looking at
- Your app provides matching content
- Results appear in system UI
Your app implements:
IntentValueQueryto receive search requestsAppEntitytypes for searchable content- Display representations for results
Platform Availability (WWDC26 297)
- Visual Intelligence runs on iOS, iPadOS, and macOS — the same entities, query, and OpenIntent code works unchanged on all three. Handle both camera captures of physical objects (iOS) and screenshots of digital media (iPad/Mac) as input.
- On Mac, the input pixel buffer can be much larger than what you'd encounter on iPhone — consider whether resizing is necessary before matching.
Quick Start
1. Import Frameworks
import VisualIntelligence
import AppIntents
2. Create App Entity
struct ProductEntity: AppEntity {
var id: String
var name: String
var price: String
var imageName: String
static var typeDisplayRepresentation: TypeDisplayRepresentation {
TypeDisplayRepresentation(
name: LocalizedStringResource("Product"),
numericFormat: "\(placeholder: .int) products"
)
}
var displayRepresentation: DisplayRepresentation {
DisplayRepresentation(
title: "\(name)",
subtitle: "\(price)",
image: .init(named: imageName)
)
}
// Deep link URL
var appLinkURL: URL? {
URL(string: "myapp://product/\(id)")
}
}
3. Create Intent Value Query
struct ProductIntentValueQuery: IntentValueQuery {
func values(for input: SemanticContentDescriptor) async throws -> [ProductEntity] {
// Search using labels
if !input.labels.isEmpty {
return await searchProducts(matching: input.labels)
}
// Search using image
if let pixelBuffer = input.pixelBuffer {
return await searchProducts(from: pixelBuffer)
}
return []
}
private func searchProducts(matching labels: [String]) async -> [ProductEntity] {
// Search your database using provided labels
// Return matching products
}
private func searchProducts(from pixelBuffer: CVReadOnlyPixelBuffer) async -> [ProductEntity] {
// Use image recognition on the pixel buffer
// Return matching products
}
}
SemanticContentDescriptor
The system provides this object with information about what the user is looking at.
Properties
| Property | Type | Description |
|---|---|---|
labels | [String] | Classification labels from Visual Intelligence |
pixelBuffer | CVReadOnlyPixelBuffer? | Raw image data |
Usage Patterns
Label-based Search:
func values(for input: SemanticContentDescriptor) async throws -> [ProductEntity] {
// Labels like "shoe", "sneaker", "Nike" etc.
let labels = input.labels
// Search your content using these labels
return products.filter { product in
labels.contains { label in
product.tags.contains(label.lowercased())
}
}
}
Image-based Search:
func values(for input: SemanticContentDescriptor) async throws -> [ProductEntity] {
guard let pixelBuffer = input.pixelBuffer else {
return []
}
// Convert to CGImage for processing
let ciImage = CIImage(cvPixelBuffer: pixelBuffer)
let context = CIContext()
guard let cgImage = context.createCGImage(ciImage, from: ciImage.extent) else {
return []
}
// Use your ML model or image matching logic
return await imageSearch.findMatches(for: cgImage)
}
Multiple Result Types
Use @UnionValue when your app has different content types.
Rules (WWDC26 297):
- An app can have only ONE
IntentValueQuerythat accepts aSemanticContentDescriptor. All result types must flow through that single query — a@UnionValueenum with one case per entity type. - Every entity type in the union needs its own
OpenIntent— without one, results of that type can't appear in image search. - Think beyond pixel matching: an album matched by image similarity can also surface the artist's nearby concerts — be creative about the type of content you return based on the context.
@UnionValue
enum SearchResult {
case product(ProductEntity)
case category(CategoryEntity)
case store(StoreEntity)
}
struct VisualSearchQuery: IntentValueQuery {
func values(for input: SemanticContentDescriptor) async throws -> [SearchResult] {
var results: [SearchResult] = []
// Search products
let products = await productSearch(input.labels)
results.append(contentsOf: products.map { .product($0) })
// Search categories
let categories = await categorySearch(input.labels)
results.append(contentsOf: categories.map { .category($0) })
return results
}
}
Display Representations
Create compelling visual representations for search results.
Result Card Real Estate (WWDC26 297)
- The search-result card gives about three lines of text for a title and subtitle, plus a thumbnail image — put the most im