跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
搜索与研究

visual-intelligence

Integrate your app with iOS Visual Intelligence for camera-based search and object recognition. Use when adding visual search capabilities.

rshankras · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add rshankras/claude-code-apple-skills --skill visual-intelligence

技能说明


name: visual-intelligence description: Integrate your app with iOS Visual Intelligence for camera-based search and object recognition. Use when adding visual search capabilities. allowed-tools: [Read, Write, Edit, Glob, Grep, Bash, AskUserQuestion] last_verified: 2026-07-16 review_by: 2027-06-22

Visual Intelligence

Integrate your app with iOS Visual Intelligence to let users find app content by pointing their camera at objects.

When This Skill Activates

  • User wants camera-based search in their app
  • User asks about visual search integration
  • User wants to surface app content in system searches
  • User needs to handle visual intelligence queries

Overview

Visual Intelligence lets users:

  1. Point camera at objects or use screenshots
  2. System identifies what they're looking at
  3. Your app provides matching content
  4. Results appear in system UI

Your app implements:

  • IntentValueQuery to receive search requests
  • AppEntity types for searchable content
  • Display representations for results

Platform Availability (WWDC26 297)

  • Visual Intelligence runs on iOS, iPadOS, and macOS — the same entities, query, and OpenIntent code works unchanged on all three. Handle both camera captures of physical objects (iOS) and screenshots of digital media (iPad/Mac) as input.
  • On Mac, the input pixel buffer can be much larger than what you'd encounter on iPhone — consider whether resizing is necessary before matching.

Quick Start

1. Import Frameworks

import VisualIntelligence
import AppIntents

2. Create App Entity

struct ProductEntity: AppEntity {
    var id: String
    var name: String
    var price: String
    var imageName: String

    static var typeDisplayRepresentation: TypeDisplayRepresentation {
        TypeDisplayRepresentation(
            name: LocalizedStringResource("Product"),
            numericFormat: "\(placeholder: .int) products"
        )
    }

    var displayRepresentation: DisplayRepresentation {
        DisplayRepresentation(
            title: "\(name)",
            subtitle: "\(price)",
            image: .init(named: imageName)
        )
    }

    // Deep link URL
    var appLinkURL: URL? {
        URL(string: "myapp://product/\(id)")
    }
}

3. Create Intent Value Query

struct ProductIntentValueQuery: IntentValueQuery {
    func values(for input: SemanticContentDescriptor) async throws -> [ProductEntity] {
        // Search using labels
        if !input.labels.isEmpty {
            return await searchProducts(matching: input.labels)
        }

        // Search using image
        if let pixelBuffer = input.pixelBuffer {
            return await searchProducts(from: pixelBuffer)
        }

        return []
    }

    private func searchProducts(matching labels: [String]) async -> [ProductEntity] {
        // Search your database using provided labels
        // Return matching products
    }

    private func searchProducts(from pixelBuffer: CVReadOnlyPixelBuffer) async -> [ProductEntity] {
        // Use image recognition on the pixel buffer
        // Return matching products
    }
}

SemanticContentDescriptor

The system provides this object with information about what the user is looking at.

Properties

PropertyTypeDescription
labels[String]Classification labels from Visual Intelligence
pixelBufferCVReadOnlyPixelBuffer?Raw image data

Usage Patterns

Label-based Search:

func values(for input: SemanticContentDescriptor) async throws -> [ProductEntity] {
    // Labels like "shoe", "sneaker", "Nike" etc.
    let labels = input.labels

    // Search your content using these labels
    return products.filter { product in
        labels.contains { label in
            product.tags.contains(label.lowercased())
        }
    }
}

Image-based Search:

func values(for input: SemanticContentDescriptor) async throws -> [ProductEntity] {
    guard let pixelBuffer = input.pixelBuffer else {
        return []
    }

    // Convert to CGImage for processing
    let ciImage = CIImage(cvPixelBuffer: pixelBuffer)
    let context = CIContext()

    guard let cgImage = context.createCGImage(ciImage, from: ciImage.extent) else {
        return []
    }

    // Use your ML model or image matching logic
    return await imageSearch.findMatches(for: cgImage)
}

Multiple Result Types

Use @UnionValue when your app has different content types.

Rules (WWDC26 297):

  • An app can have only ONE IntentValueQuery that accepts a SemanticContentDescriptor. All result types must flow through that single query — a @UnionValue enum with one case per entity type.
  • Every entity type in the union needs its own OpenIntent — without one, results of that type can't appear in image search.
  • Think beyond pixel matching: an album matched by image similarity can also surface the artist's nearby concerts — be creative about the type of content you return based on the context.
@UnionValue
enum SearchResult {
    case product(ProductEntity)
    case category(CategoryEntity)
    case store(StoreEntity)
}

struct VisualSearchQuery: IntentValueQuery {
    func values(for input: SemanticContentDescriptor) async throws -> [SearchResult] {
        var results: [SearchResult] = []

        // Search products
        let products = await productSearch(input.labels)
        results.append(contentsOf: products.map { .product($0) })

        // Search categories
        let categories = await categorySearch(input.labels)
        results.append(contentsOf: categories.map { .category($0) })

        return results
    }
}

Display Representations

Create compelling visual representations for search results.

Result Card Real Estate (WWDC26 297)

  • The search-result card gives about three lines of text for a title and subtitle, plus a thumbnail image — put the most im
阅读完整来源 ↗