ai-foundry-posture-check
Assess security, reliability, and cost posture of Azure OpenAI and Microsoft Foundry deployments. Detects critical anti-patterns: API keys instead of Managed Identity, public endpoints without firewall, disabled content filtering, missing diagnostic settings, deprecated model versions, over-provisioned PTU, absence of AI Gateway (APIM) for rate limiting, shared dev/prod deployments, quota exhaustion risk, 429 throttling rates, runaway token consumption, and single-region deployment. Use when asked about AI workload security, OpenAI best practices, Foundry health check, model deployment review, AI cost optimization, quota monitoring, or throttling analysis.
ricmmartins · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add ricmmartins/azure-sre-agent-skills --skill ai-foundry-posture-check技能说明
name: ai-foundry-posture-check description: > Assess security, reliability, and cost posture of Azure OpenAI and Microsoft Foundry deployments. Detects critical anti-patterns: API keys instead of Managed Identity, public endpoints without firewall, disabled content filtering, missing diagnostic settings, deprecated model versions, over-provisioned PTU, absence of AI Gateway (APIM) for rate limiting, shared dev/prod deployments, quota exhaustion risk, 429 throttling rates, runaway token consumption, and single-region deployment. Use when asked about AI workload security, OpenAI best practices, Foundry health check, model deployment review, AI cost optimization, quota monitoring, or throttling analysis. tools:
- RunAzCliReadCommands
- execute_kusto_query
AI Foundry & OpenAI Posture Check
Purpose
Assess the security, reliability, and cost efficiency of Azure OpenAI and Microsoft Foundry deployments. Detects the most common anti-patterns that startups make when building AI-powered products — from exposed endpoints to runaway token costs.
Based on the Azure Well-Architected Framework for AI workloads and Microsoft Foundry operational best practices.
When to use this skill
- User asks "is our OpenAI deployment secure?"
- User asks about AI cost optimization or token consumption
- Review before going to production with an AI feature
- User asks about content filtering, model versions, or rate limiting
- Periodic AI workload health check
Pre-check
Confirm with the user:
- Which Azure OpenAI / Cognitive Services accounts to assess (or "all in subscription")
- Whether they use PTU (Provisioned Throughput) or Standard deployments
- Whether they have production AI workloads already live
Assessment procedure
Step 0: Discover AI resources
az cognitiveservices account list \
--subscription <sub-id> \
--query "[?kind=='OpenAI' || kind=='AIServices'].{name:name, kind:kind, rg:resourceGroup, location:location, sku:sku.name}" \
-o table
If no results, try:
az cognitiveservices account list \
--subscription <sub-id> \
--query "[].{name:name, kind:kind, rg:resourceGroup, location:location}" \
-o table
If no Cognitive Services accounts exist, report "No Azure OpenAI or AI Foundry resources found" and end assessment.
For each account found, run the following checks:
🔐 CATEGORY 1 — Security (Critical)
Check 1.1 — Managed Identity enabled (not API keys only)
az cognitiveservices account show \
--name <account> --resource-group <rg> \
--query "{identity:identity.type, disableLocalAuth:properties.disableLocalAuth}" \
-o json
| Finding | Severity | Score |
|---|---|---|
identity.type = SystemAssigned/UserAssigned AND disableLocalAuth = true | ✅ Pass | 12 pts |
identity.type set but disableLocalAuth = false | ⚠️ Partial | 6 pts |
identity.type = None or null | ❌ Fail | 0 pts |
📖 https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/managed-identity
Check 1.2 — Network isolation (firewall or private endpoint)
az cognitiveservices account show \
--name <account> --resource-group <rg> \
--query "{publicAccess:properties.publicNetworkAccess, defaultAction:properties.networkAcls.defaultAction, privateEndpoints:properties.privateEndpointConnections}" \
-o json
| Finding | Severity | Score |
|---|---|---|
publicNetworkAccess = Disabled + private endpoints exist | ✅ Pass | 13 pts |
defaultAction = Deny + IP/VNet rules configured | ✅ Pass | 13 pts |
defaultAction = Allow (open to internet) | ❌ Fail | 0 pts |
📖 https://learn.microsoft.com/en-us/azure/ai-services/cognitive-services-virtual-networks
Check 1.3 — Content filtering (RAI policies)
az rest --method get \
--url "https://management.azure.com/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<account>/raiPolicies?api-version=2024-10-01"
Then check deployments have a policy assigned:
az cognitiveservices account deployment list \
--name <account> --resource-group <rg> \
--query "[].{name:name, model:properties.model.name, raiPolicy:properties.raiPolicyName}" \
-o table
| Finding | Severity | Score |
|---|---|---|
| All deployments have RAI policy assigned with filters enabled | ✅ Pass | 10 pts |
| Some deployments missing policy or using minimal filters | ⚠️ Partial | 5 pts |
| No RAI policies or content filtering disabled | ❌ Fail | 0 pts |
📖 https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/content-filters
⚙️ CATEGORY 2 — Reliability & Operations
Check 2.1 — Model version pinned and not deprecated
az cognitiveservices account deployment list \
--name <account> --resource-group <rg> \
--query "[].{name:name, model:properties.model.name, version:properties.model.version}" \
-o table
Cross-reference with available models:
az cognitiveservices account list-models \
--name <account> --resource-group <rg> \
--query "[].{model:model.name, version:model.version, lifecycle:model.lifecycleStatus}" \
-o table
| Finding | Severity | Score |
|---|---|---|
| All deployments on GA versions with explicit version pinned | ✅ Pass | 7 pts |
| Deployments on versions nearing retirement (< 3 months) | ⚠️ Attention | 3 pts |
| Deployments on Legacy/Deprecated versions | ❌ Fail | 0 pts |
📖 https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/model-retirements
Check 2.2 — Diagnostic settings enabled
# Get resource ID
RESOURCE_ID=$(az cognitiveservices account show \
--name <account> --resource-group <rg> --query "id" -o tsv)
az monitor diagnostic-settings list --resource $RESOURCE_ID -o json
| Finding | Severity | Score |
|---|---|---|
| Diagnostic settings sending RequestResponse + Audit to Log Analytics | ✅ Pass |