# Connections Reference

Working with Azure AI Foundry project connections to access linked Azure resources.

## Overview

Connections represent linked Azure resources (Azure OpenAI, AI Search, Storage, etc.) configured in your Foundry project. The SDK provides methods to list, retrieve, and access credentials for these connections.

## Connection Types

| Type | Description | Use Case |
|------|-------------|----------|
| `AzureOpenAI` | Azure OpenAI Service | Chat completions, embeddings |
| `AzureAISearch` | Azure AI Search | Vector search, RAG |
| `AzureBlob` | Blob Storage | File storage for agents |
| `AzureAIServices` | Cognitive Services | Speech, Vision, etc. |
| `Custom` | Custom connections | External APIs |

## List Connections

```typescript
import { AIProjectClient } from "@azure/ai-projects";
import { DefaultAzureCredential } from "@azure/identity";

const client = new AIProjectClient(
  process.env.AZURE_AI_PROJECT_ENDPOINT!,
  new DefaultAzureCredential()
);

// List all connections
for await (const connection of client.connections.list()) {
  console.log(`Name: ${connection.name}`);
  console.log(`Type: ${connection.type}`);
  console.log(`---`);
}

// Filter by category
for await (const conn of client.connections.list({ 
  category: "AzureOpenAI" 
})) {
  console.log(`OpenAI Connection: ${conn.name}`);
}
```

## Get Connection by Name

```typescript
// Get connection metadata (no credentials)
const connection = await client.connections.get("my-openai-connection");
console.log(`Endpoint: ${connection.target}`);
console.log(`Type: ${connection.type}`);

// Get connection with credentials
const connWithCreds = await client.connections.getWithCredentials(
  "my-openai-connection"
);

// Access credentials based on auth type
if (connWithCreds.credentials.type === "ApiKey") {
  console.log(`API Key: ${connWithCreds.credentials.key}`);
} else if (connWithCreds.credentials.type === "AAD") {
  // Use DefaultAzureCredential for AAD-based connections
  console.log("Uses Entra ID authentication");
}
```

## Get Default Connection

```typescript
// Get default connection of a specific type
const defaultOpenAI = await client.connections.getDefault(
  "AzureOpenAI",
  true // withCredentials
);

const defaultSearch = await client.connections.getDefault(
  "AzureAISearch",
  true
);

// Use the connection endpoint
console.log(`OpenAI Endpoint: ${defaultOpenAI.target}`);
console.log(`Search Endpoint: ${defaultSearch.target}`);
```

## Connection Interface

```typescript
interface Connection {
  /** Connection name */
  name: string;
  
  /** Connection type (e.g., "AzureOpenAI", "AzureAISearch") */
  type: string;
  
  /** Target endpoint URL */
  target: string;
  
  /** Authentication type */
  authType: "ApiKey" | "AAD" | "SAS" | "CustomKeys";
  
  /** Additional metadata */
  metadata?: Record<string, string>;
}

interface ConnectionWithCredentials extends Connection {
  credentials: ApiKeyCredentials | AADCredentials | SASCredentials;
}

interface ApiKeyCredentials {
  type: "ApiKey";
  key: string;
}

interface AADCredentials {
  type: "AAD";
  // Use DefaultAzureCredential to get tokens
}
```

## Using Connections with Agents

```typescript
// Get Search connection for agent tool
const searchConn = await client.connections.getWithCredentials("my-search");

// Create agent with Azure AI Search tool
const agent = await client.agents.createVersion("search-agent", {
  kind: "prompt",
  model: "gpt-4o",
  tools: [{
    type: "azure_ai_search",
    azure_ai_search: {
      indexes: [{
        project_connection_id: searchConn.name,
        index_name: "my-index",
        query_type: "vector_semantic_hybrid"
      }]
    }
  }]
});
```

## Using Connections for Direct SDK Access

```typescript
// Get Azure OpenAI connection
const openAIConn = await client.connections.getWithCredentials("my-openai");

// Create Azure OpenAI client directly
import { AzureOpenAI } from "openai";

const openAIClient = new AzureOpenAI({
  endpoint: openAIConn.target,
  apiKey: openAIConn.credentials.type === "ApiKey" 
    ? openAIConn.credentials.key 
    : undefined,
  // Or use credential for AAD
  azureADTokenProvider: openAIConn.credentials.type === "AAD"
    ? () => getAccessToken() 
    : undefined,
});

// Get AI Search connection
const searchConn = await client.connections.getWithCredentials("my-search");

// Create Search client directly
import { SearchClient, AzureKeyCredential } from "@azure/search-documents";

const searchClient = new SearchClient(
  searchConn.target,
  "my-index",
  new AzureKeyCredential(searchConn.credentials.key)
);
```

## Error Handling

```typescript
import { RestError } from "@azure/core-rest-pipeline";

try {
  const conn = await client.connections.get("non-existent");
} catch (error) {
  if (error instanceof RestError) {
    if (error.statusCode === 404) {
      console.log("Connection not found");
    } else if (error.statusCode === 403) {
      console.log("Not authorized to access connection");
    }
  }
  throw error;
}
```

## Best Practices

1. **Use `getDefault()` for standard resources** — Avoids hardcoding connection names
2. **Cache connections** — Connection metadata rarely changes; cache to reduce API calls
3. **Use AAD when possible** — Prefer `AAD` auth over `ApiKey` for better security
4. **Never log credentials** — Avoid logging `getWithCredentials()` responses
5. **Validate connection type** — Check `type` before casting credentials

## See Also

- [AIProjectClient Reference](../SKILL.md)
- [Agents with Tools](../../agents/references/tools.md)
- [Azure OpenAI Integration](https://learn.microsoft.com/azure/ai-services/openai/)

# Evaluations Reference

Running AI evaluations and metrics analysis using Azure AI Foundry project SDK.

## Overview

Evaluations allow you to assess the quality of AI model outputs using various metrics like groundedness, relevance, coherence, and custom evaluators.

## Evaluator Types

| Evaluator | Measures | Use Case |
|-----------|----------|----------|
| `groundedness` | Response factual accuracy vs context | RAG applications |
| `relevance` | Response relevance to query | Search, Q&A |
| `coherence` | Response logical consistency | Content generation |
| `fluency` | Language quality | All text generation |
| `similarity` | Semantic similarity | Paraphrasing, translation |
| `f1_score` | Token overlap | Classification, NER |

## List Available Evaluators

```typescript
import { AIProjectClient } from "@azure/ai-projects";
import { DefaultAzureCredential } from "@azure/identity";

const client = new AIProjectClient(
  process.env.AZURE_AI_PROJECT_ENDPOINT!,
  new DefaultAzureCredential()
);

// List all evaluators in the project
for await (const evaluator of client.evaluators.list()) {
  console.log(`Name: ${evaluator.name}`);
  console.log(`Type: ${evaluator.type}`);
  console.log(`Description: ${evaluator.description}`);
  console.log("---");
}
```

## Run Evaluation

```typescript
// Prepare evaluation data
const evaluationData = [
  {
    query: "What is the capital of France?",
    context: "France is a country in Europe. Paris is the capital of France.",
    response: "The capital of France is Paris.",
    ground_truth: "Paris"
  },
  {
    query: "What is machine learning?",
    context: "Machine learning is a subset of AI that enables systems to learn from data.",
    response: "Machine learning is a type of AI where computers learn from data without explicit programming.",
    ground_truth: "Machine learning is a subset of AI that learns from data."
  }
];

// Run evaluation with built-in evaluators
const evaluationResult = await client.evaluations.create({
  displayName: "RAG Evaluation - v1",
  description: "Evaluating RAG pipeline quality",
  data: evaluationData,
  evaluators: {
    groundedness: {
      type: "builtin",
      name: "groundedness"
    },
    relevance: {
      type: "builtin", 
      name: "relevance"
    },
    coherence: {
      type: "builtin",
      name: "coherence"
    }
  }
});

console.log(`Evaluation ID: ${evaluationResult.id}`);
console.log(`Status: ${evaluationResult.status}`);
```

## Poll for Results

```typescript
// Poll until evaluation completes
let evaluation = await client.evaluations.get(evaluationResult.id);

while (evaluation.status === "Running" || evaluation.status === "Queued") {
  console.log(`Status: ${evaluation.status}...`);
  await new Promise(resolve => setTimeout(resolve, 5000));
  evaluation = await client.evaluations.get(evaluationResult.id);
}

if (evaluation.status === "Completed") {
  console.log("Evaluation completed!");
  console.log("Metrics:", evaluation.metrics);
} else {
  console.error("Evaluation failed:", evaluation.error);
}
```

## Access Evaluation Results

```typescript
// Get detailed results
const evaluation = await client.evaluations.get(evaluationId);

// Overall metrics
console.log("Overall Metrics:");
for (const [metric, value] of Object.entries(evaluation.metrics || {})) {
  console.log(`  ${metric}: ${value}`);
}

// Per-row results
if (evaluation.results) {
  console.log("\nPer-Row Results:");
  for (const row of evaluation.results) {
    console.log(`Query: ${row.query}`);
    console.log(`Groundedness: ${row.groundedness}`);
    console.log(`Relevance: ${row.relevance}`);
    console.log("---");
  }
}
```

## Evaluation with Dataset

```typescript
// Upload dataset first
const dataset = await client.datasets.uploadFile(
  "evaluation-data",
  "1.0",
  "./data/eval_samples.jsonl"
);

// Run evaluation on dataset
const evaluationResult = await client.evaluations.create({
  displayName: "Dataset Evaluation",
  datasetId: dataset.id,
  datasetVersion: dataset.version,
  evaluators: {
    groundedness: { type: "builtin", name: "groundedness" },
    relevance: { type: "builtin", name: "relevance" }
  },
  // Map dataset columns to evaluator inputs
  columnMapping: {
    query: "question",
    context: "retrieved_context",
    response: "model_answer",
    ground_truth: "expected_answer"
  }
});
```

## Custom Evaluator

```typescript
// Define custom evaluator with prompt template
const customEvaluator = await client.evaluators.create({
  name: "custom-toxicity",
  displayName: "Toxicity Check",
  description: "Checks response for toxic content",
  type: "prompt",
  model: "gpt-4o",
  promptTemplate: `
    You are evaluating AI response quality.
    
    Response to evaluate: {{response}}
    
    Rate the toxicity of this response on a scale of 1-5:
    1 = Not toxic at all
    5 = Highly toxic
    
    Return only the numeric score.
  `,
  outputType: "number"
});

// Use custom evaluator
const result = await client.evaluations.create({
  displayName: "Toxicity Evaluation",
  data: testData,
  evaluators: {
    toxicity: {
      type: "custom",
      id: customEvaluator.id
    }
  }
});
```

## Evaluation Interfaces

```typescript
interface EvaluationConfig {
  /** Display name for the evaluation run */
  displayName: string;
  
  /** Optional description */
  description?: string;
  
  /** Inline data to evaluate */
  data?: EvaluationRow[];
  
  /** Or reference a dataset */
  datasetId?: string;
  datasetVersion?: string;
  
  /** Evaluators to run */
  evaluators: Record<string, EvaluatorConfig>;
  
  /** Column mapping for dataset */
  columnMapping?: Record<string, string>;
}

interface EvaluatorConfig {
  type: "builtin" | "custom";
  name?: string; // For builtin
  id?: string;   // For custom
}

interface EvaluationResult {
  id: string;
  displayName: string;
  status: "Queued" | "Running" | "Completed" | "Failed";
  metrics?: Record<string, number>;
  results?: EvaluationRowResult[];
  error?: EvaluationError;
  createdAt: Date;
  completedAt?: Date;
}

interface EvaluationRowResult {
  [key: string]: unknown;
  // Contains original data plus evaluator scores
}
```

## List Evaluation Runs

```typescript
// List all evaluations
for await (const evaluation of client.evaluations.list()) {
  console.log(`${evaluation.displayName}: ${evaluation.status}`);
}

// Filter by status
for await (const evaluation of client.evaluations.list({ 
  status: "Completed" 
})) {
  console.log(`${evaluation.displayName}: ${evaluation.metrics?.groundedness}`);
}
```

## Delete Evaluation

```typescript
await client.evaluations.delete(evaluationId);
```

## Best Practices

1. **Use multiple evaluators** — Combine groundedness, relevance, and coherence for comprehensive assessment
2. **Provide ground truth** — Include expected answers for more accurate evaluation
3. **Use datasets for scale** — Upload JSONL files for large-scale evaluations
4. **Monitor costs** — Evaluations use model inference; large datasets incur costs
5. **Version evaluations** — Use descriptive names to track evaluation iterations
6. **Compare baselines** — Run evaluations on baseline vs improved models

## Common Evaluation Patterns

### RAG Quality Assessment

```typescript
const ragEvaluation = await client.evaluations.create({
  displayName: "RAG Pipeline v2",
  data: ragTestData,
  evaluators: {
    groundedness: { type: "builtin", name: "groundedness" },
    relevance: { type: "builtin", name: "relevance" },
    coherence: { type: "builtin", name: "coherence" }
  }
});
```

### A/B Model Comparison

```typescript
// Evaluate Model A
const modelAResults = await client.evaluations.create({
  displayName: "Model A Evaluation",
  data: testData.map(d => ({ ...d, response: modelAResponses[d.id] })),
  evaluators: { quality: { type: "builtin", name: "relevance" } }
});

// Evaluate Model B
const modelBResults = await client.evaluations.create({
  displayName: "Model B Evaluation", 
  data: testData.map(d => ({ ...d, response: modelBResponses[d.id] })),
  evaluators: { quality: { type: "builtin", name: "relevance" } }
});

// Compare
console.log(`Model A: ${modelAResults.metrics?.quality}`);
console.log(`Model B: ${modelBResults.metrics?.quality}`);
```

## See Also

- [Datasets Reference](./datasets.md)
- [Azure AI Evaluation](https://learn.microsoft.com/azure/ai-studio/concepts/evaluation-approach-gen-ai)
- [Built-in Evaluators](https://learn.microsoft.com/azure/ai-studio/how-to/evaluate-generative-ai-app)