Claude Cursor GitHub Copilot Skill

dotnet-trace-collect

Guide developers through capturing diagnostic artifacts to diagnose production .NET performance issues. Use when the user needs help choosing diagnostic tools, collecting performance data, or understanding tool trade-offs across different environments (Windows/Linux, .NET Framewo

LLM Mart · 0 points · 20 views 0 listing impressions 0 install-command copies
Virus-scanned Reviewed automatically before listing.

Full trust report

Download dotnet-skills-plugins_dotnet-diag_skills_dotnet-trace-collect-98f8485.zip · 16 KB
dotnet/skills 5471 418 forks MIT Updated 1d ago
Part of dotnet/skills — 119 skills

Install

skills CLI npx skills add https://github.com/dotnet/skills/tree/main/plugins/dotnet-diag/skills/dotnet-trace-collect
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install dotnet-skills@llmmart
Git git clone https://github.com/dotnet/skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole dotnet/skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

.NET Trace Collect

This skill helps developers diagnose production performance issues by recommending the right diagnostic tools for their environment, guiding data collection, and suggesting analysis approaches. It does not analyze code for anti-patterns or perform the analysis itself.

When to Use

  • A developer needs to investigate a production performance issue (high CPU, memory leak, slow requests, excessive GC, networking errors, etc.)
  • Choosing the right diagnostic tool for a specific runtime, OS, or deployment topology
  • Setting up and running diagnostic tool commands for data collection
  • Understanding trade-offs between available tools (e.g. PerfView vs dotnet-trace)
  • Collecting diagnostics from containerized or Kubernetes workloads

When Not to Use

  • Reviewing source code for performance anti-patterns (use a code review skill instead)
  • Benchmarking during development (e.g. BenchmarkDotNet setup)
  • Analyzing collected trace or dump files (this skill recommends tools for analysis, but does not perform it)

Inputs

Input Required Description
Symptom Yes What the developer is observing (high CPU, memory growth, slow requests, hangs, excessive GC, HTTP 5xx errors, networking timeouts, connection failures, assembly loading failures, etc.)
Runtime Yes .NET Framework or modern .NET (and version, especially whether .NET 10+)
OS Yes Windows or Linux
Deployment Yes Non-container, container, or Kubernetes
Admin privileges Recommended Whether the developer has admin/root access on the target machine
Repro characteristics Recommended Whether the issue is easy to reproduce or requires a long time to manifest

Workflow

Step 1: Understand the environment

Determine or ask the developer to clarify:

  1. Symptom: What they are observing (high CPU, memory leak, slow requests, hangs, excessive GC, HTTP 5xx errors, networking timeouts, connection failures, assembly loading failures, etc.)
  2. Runtime: .NET Framework or modern .NET? If modern .NET, which version? (Especially whether .NET 10 or later.)
  3. OS: Windows or Linux?
  4. Deployment: Running directly on the host, in a container, or in Kubernetes?
  5. Admin privileges: Do they have admin/root access on the target machine or container?
  6. Repro characteristics: Does the issue reproduce quickly, or does it take a long time to manifest?
  7. Workload context: Determine or ask the user if you are running in the context of the workload (i.e., on the same machine or connected to the same environment where the issue is occurring). If so, you can run diagnostic commands directly on their behalf. If not, provide the commands as guidance for the user to run themselves.

Use this information to select the right tool in Step 2.

Step 2: Recommend diagnostic tools

Select tools based on the environment using the priority rules below. Once a tool is selected, load the corresponding reference file for detailed command-line usage.

Tool reference lookup

Environment Reference file(s)
Windows + modern .NET + admin references/perfview.md
Windows + modern .NET, no admin references/dotnet-trace-collect.md
Windows + .NET Framework references/perfview.md
Linux + .NET 10+ + root references/dotnet-trace-collect-linux.md
Linux + pre-.NET 10 references/dotnet-trace-collect.md
Linux + native stacks needed references/perfcollect.md
Container/K8s (console access) references/dotnet-trace-collect.md (or dotnet-trace-collect-linux.md)
Container/K8s (no console) references/dotnet-monitor.md

Quick decision matrix (first-pass triage)

Environment Preferred tool Fallback / Notes
Windows + modern .NET + admin PerfView If admin is unavailable, use dotnet-trace
Windows + .NET Framework + admin PerfView Without admin, there is no trace fallback; for hangs/memory leaks, provide dump commands directly (procdump -ma or Task Manager) since dump-collect does not support .NET Framework
Linux + .NET 10+ + root dotnet-trace collect-linux Use dotnet-trace if root or kernel prerequisites are not met
Linux + pre-.NET 10 dotnet-trace Add perfcollect when native stacks are needed (requires root)
Linux container/Kubernetes Console tools if in workload context; dotnet-monitor if no console access See Linux Container / Kubernetes section for details

Windows (non-container, modern .NET)

  1. PerfView (preferred) — produces richer ETW-based data; requires admin privileges. For slow requests, add /ThreadTime to capture thread-level wait and block detail.
  2. dotnet-trace — fallback when admin privileges are not available.
  3. For long-running repros: use PerfView with a /StopOn trigger that fires on the symptom you want to capture (e.g., /StopOnPerfCounter, /StopOnGCEvent, /StopOnException) and a circular buffer (/CircularMB + /BufferSizeMB). Critical: the stop trigger must fire on the interesting event, not the recovery. The circular buffer continuously overwrites old data, so if you trigger on recovery, the buffer may have already overwritten the interesting behavior by the time collection stops. Only add /StartOn if the start event is known to precede the stop event. For slow requests, do not include a stop trigger by default — let the user design one based on their specific scenario.

Windows containers

  1. PerfView — most Windows containers (including Kubernetes on Windows) use process-isolation by default. Collect from the host with /EnableEventsInContainers. After collection, you have two options:

    • Analyze locally while the container is still running — PerfView can reach into the live container to resolve symbols, so you can open the trace immediately on the host machine.
    • Analyze off-machine — before the container shuts down, copy the .etl.zip into the container and run PerfViewCollect merge /ImageIDsOnly inside it to embed symbol information. Then copy the merged trace out. Without this merge step, symbols for binaries inside the container will be unresolvable on other machines.

    For the less common Hyper-V containers, collect inside the container directly. See references/perfview.md for detailed commands.

  2. dotnet-monitor, dotnet-trace — inside the container if the tools are installed in the image. For dumps, invoke the dump-collect skill.

Windows (.NET Framework)

  1. PerfView — the primary diagnostic tool for .NET Framework on Windows. Requires admin.
  2. Same trigger guidance for long repros: use /StopOn triggers that fire on the symptom (e.g., /StopOnPerfCounter, /StopOnGCEvent, /StopOnException) with /CircularMB + /BufferSizeMB.
  3. Without admin: PerfView requires admin, and there are no alternative trace tools for .NET Framework. Process dumps can still be captured without admin — provide dump commands directly (e.g., procdump -ma <PID> or Task Manager) since the dump-collect skill does not support .NET Framework. Dumps can help diagnose hangs and memory leaks. However, for high CPU, slow requests, and excessive GC, there is no way to investigate on .NET Framework without admin access. Advise the user to obtain admin privileges.

Linux (non-container, .NET 10+)

  1. dotnet-trace collect-linux (preferred) — uses perf_events for richer traces including native call stacks and kernel events. Captures machine-wide by default (no PID required). Requires root and kernel >= 6.4.
  2. dotnet-trace — fallback when root privileges are not available or kernel requirements are not met. Managed stacks only.

Linux (non-container, pre-.NET 10)

  1. dotnet-trace (preferred) — managed trace collection; no admin required.
  2. perfcollect — when native call stacks are needed (requires admin/root).

Linux Container / Kubernetes

If running in the context of the workload (i.e., you have console access to the container), prefer console-based tools. These are easier to set up than dotnet-monitor, which requires authentication configuration and sidecar deployment:

  1. dotnet-trace collect-linux (.NET 10+ with root) — produces the richest traces including native call stacks and kernel events.
  2. dotnet-trace — inside the container if the tool is installed in the image. For dumps, invoke the dump-collect skill.
  3. perfcollect — inside the container when native stacks are needed on pre-.NET 10 (requires SYS_ADMIN / --privileged).

If not running in the workload context (no console access), or if dotnet-monitor is already deployed:

  1. dotnet-monitor — designed for containers; runs as a sidecar. No tools needed in the app container. Easiest option when console access is not available.

Memory dumps

When dumps are needed (memory leaks, hangs), do not provide dump collection commands directly for modern .NET — invoke the dump-collect skill instead. The dump-collect skill only supports modern .NET (.NET Core 3.0+). For .NET Framework, provide dump collection guidance directly (e.g., procdump -ma <PID> or Task Manager). This skill focuses on trace collection only.

Memory leaks

  • Capture two dumps as memory is increasing (e.g., one early, one after significant growth). Invoke the dump-collect skill for dump collection — do not provide dump commands directly. Diff the dumps in PerfView to see which objects have increased — this is the most effective way to identify what is leaking.
  • Without admin privileges: Two process dumps can give a sense of what's growing on the heap, but may not be enough to identify the root cause. If dumps aren't sufficient, reproduce the issue in an environment where admin privileges are available to collect richer data (traces).
  • Modern .NET on Linux (pre-.NET 10): Recommend two dump captures (invoke dump-collect skill) for heap diff, plus dotnet-trace while memory is growing (for allocation tracking). No trigger needed — capture during the growth period. Both together give the best picture.
  • Modern .NET 10+ on Linux with admin: Recommend two dump captures (invoke dump-collect skill) for heap diff, plus dotnet-trace collect-linux while memory is growing (richer data including native stacks). No trigger needed.
  • .NET Framework: Recommend two dumps plus a PerfView trace while memory is growing to see what is being allocated. The dump-collect skill does not support .NET Framework, so provide dump commands directly (e.g., procdump -ma <PID> or right-click → Create Dump File in Task Manager). No trigger is needed — just capture the trace during the growth period. Do not wait for an OutOfMemoryException.

Excessive GC

Excessive GC requires a trace to analyze GC events, pause times, and allocation patterns — a dump is not sufficient.

  • Windows (PerfView): Use PerfView collect /GCCollectOnly to capture GC events.
  • Linux (dotnet-trace): Use dotnet-trace collect -p <PID> --profile gc-verbose.
  • Linux .NET 10+ with root: Use dotnet-trace collect-linux --profile gc-verbose for richer data with native stacks.
  • Containers: dotnet-monitor can capture GC traces via its REST API (/trace?profile=gc-verbose).

Slow Requests

Slow requests require a thread time trace to see where threads are spending time — waiting on locks, I/O, external calls, etc. Use larger buffers since thread time traces generate more data. For ASP.NET Core applications, also enable Microsoft.AspNetCore.Hosting and Microsoft-AspNetCore-Server-Kestrel providers to get server-side request lifecycle timing (when requests arrive, how long they take to process).

  • Windows (PerfView): Use PerfView /ThreadTime collect /BufferSizeMB:1024 /CircularMB:2048. The /ThreadTime argument adds thread-level wait and block detail. For ASP.NET Core, add Kestrel providers: PerfView /ThreadTime collect /BufferSizeMB:1024 /CircularMB:2048 /Providers:*Microsoft.AspNetCore.Hosting,*Microsoft-AspNetCore-Server-Kestrel. Do not include a stop trigger by default — let the user design one based on their specific scenario.
  • Linux (dotnet-trace): dotnet-trace captures thread time data by default — no special arguments needed. Use dotnet-trace collect -p <PID>. For ASP.NET Core, add Kestrel providers: dotnet-trace collect -p <PID> --providers Microsoft.AspNetCore.Hosting,Microsoft-AspNetCore-Server-Kestrel.
  • Linux .NET 10+ with root: Use dotnet-trace collect-linux --profile thread-time for richer data with native stacks. For ASP.NET Core, add: --providers Microsoft.AspNetCore.Hosting,Microsoft-AspNetCore-Server-Kestrel.
  • Containers: dotnet-monitor can capture traces via its REST API (/trace?pid=<PID>&durationSeconds=30).

Hangs

  1. Start with a trace to understand what threads are doing. Use the appropriate trace tool for the environment (PerfView with /ThreadTime on Windows, dotnet-trace on Linux, dotnet-trace collect-linux --profile thread-time on .NET 10+ Linux with root). The trace can reveal:
    • Livelocks (threads spinning without forward progress) — threads appear busy but the application makes no progress.
    • Thread starvation — the ThreadPool is exhausted and queued work items are not being processed. This can look like a deadlock but has a different root cause.
    • Whether there is any forward progress at all — if some threads are making progress, the issue may be a bottleneck rather than a true hang.
  2. If the trace does not explain the hang, the issue may be a true deadlock (threads waiting on each other in a cycle). In this case, invoke the dump-collect skill to collect a process dump — do not provide dump commands directly.
  3. Analyze the dump with a debugger to inspect thread stacks and identify the lock cycle:
    • Windows: Visual Studio or WinDbg with the SOS debugger extension.
    • Linux: lldb with the SOS debugger extension.

Networking Issues

Networking issues (HTTP 5xx errors from downstream services, request timeouts, connection failures, DNS resolution failures, TLS handshake failures, connection pool exhaustion) require both a thread-time trace and networking event providers. The thread-time trace shows where threads are blocked (slow downstream calls, thread starvation), while the networking events show the request lifecycle — which requests failed, what status codes came back, how long DNS resolution and TLS handshakes took, and how long requests waited for a connection from the pool.

For .NET Framework, PerfView /ThreadTime already collects the relevant networking events (from the System.Net ETW provider) — no additional providers are needed.

For modern .NET, you must explicitly enable the System.Net.* EventSource providers:

Provider What it covers
System.Net.Http HttpClient/SocketsHttpHandler — request lifecycle, HTTP status codes, connection pool
System.Net.NameResolution DNS lookups (start/stop, duration)
System.Net.Security TLS/SSL handshakes (SslStream)
System.Net.Sockets Low-level socket connect/disconnect

Key events from System.Net.Http: RequestStart (scheme, host, port, path), RequestStop (statusCode — -1 if no response was received), RequestFailed (exception message for timeouts, connection refused, etc.), RequestLeftQueue (time waiting for a connection from the pool — indicates connection pool exhaustion), ConnectionEstablished, ConnectionClosed.

Collect a thread-time trace with networking providers enabled (modern .NET only — .NET Framework needs only PerfView /ThreadTime):

  • Windows (PerfView): Use PerfView /ThreadTime collect /BufferSizeMB:1024 /CircularMB:2048 /Providers:*System.Net.Http,*System.Net.NameResolution,*System.Net.Security,*System.Net.Sockets. For .NET Framework, omit the /Providers flag — /ThreadTime already includes the networking events. The thread-time trace shows where threads are blocked while the networking events show what requests are failing and why.
  • Linux (dotnet-trace): dotnet-trace captures thread time data by default, but specifying --providers overrides the defaults so you must also include --profile: dotnet-trace collect -p <PID> --profile dotnet-common,dotnet-sampled-thread-time --providers System.Net.Http,System.Net.NameResolution,System.Net.Security,System.Net.Sockets.
  • Linux .NET 10+ with root: Use dotnet-trace collect-linux --profile dotnet-common,cpu-sampling,thread-time --providers System.Net.Http,System.Net.NameResolution,System.Net.Security,System.Net.Sockets.
  • Containers: dotnet-monitor can capture traces with custom providers via its REST API.

Assembly Loading Issues

For modern .NET, assembly loading issues (FileNotFoundException, FileLoadException, ReflectionTypeLoadException, version conflicts, duplicate assembly loads across AssemblyLoadContexts) require collecting assembly loader binder events from the Microsoft-Windows-DotNETRuntime provider with the Loader keyword (0x4). These events trace every step of the runtime's assembly resolution algorithm — which paths were probed, which AssemblyLoadContext handled the load, whether the load succeeded or failed, and why. For .NET Framework, the same provider and keyword work for ETW-based collection; additionally, the Fusion Log Viewer (fuslogvw.exe) can diagnose assembly binding failures without requiring a trace.

The provider specification is Microsoft-Windows-DotNETRuntime:0x4:4 (provider name, AssemblyLoader keyword, Informational verbosity).

  • Windows (PerfView): A default PerfView trace already includes binder events - simply run PerfView collect with no extra providers. For a smaller trace file, use PerfView collect /ClrEvents:Default-Profile, which removes the most verbose default events while keeping the events necessary for diagnosing assembly loading issues.
  • Linux / cross-platform (dotnet-trace): Use dotnet-trace collect --clrevents assemblyloader -- <path-to-built-exe> to launch and trace the process, or dotnet-trace collect --clrevents assemblyloader -p <PID> to attach to a running process.
  • Linux .NET 10+ with root: Use dotnet-trace collect-linux --clrevents assemblyloader.
  • Containers: dotnet-monitor can capture traces with the loader provider via its REST API.

For short-lived processes that fail on startup (common with assembly loading issues), prefer the dotnet-trace launch form (-- <path-to-built-exe>) over attaching by PID, since the process may exit before you can attach.

Explain the trade-offs when recommending a tool. For example:

  • PerfView gives richer data but needs admin; runs on Windows including Windows containers.
  • dotnet-trace works cross-platform without admin but captures less system-level detail.
  • perfcollect captures native call stacks but needs admin/root.
  • dotnet-monitor is the best option for containers/K8s when console access is not available, but requires sidecar deployment and authentication configuration.

Step 3: Guide data collection

Provide the specific commands for the recommended tool. Load the appropriate reference file from the tool reference lookup table for detailed command-line examples.

Key guidance to include:

  1. Installation: How to install the tool if it is not already available (e.g. dotnet tool install -g dotnet-trace). When recommending multiple tools, provide installation and usage instructions for each one — do not mention a tool without showing how to install and use it.
  2. PID discovery (required before any -p <PID> command): Verify the target process first (for example: dotnet-trace ps, curl <monitor-endpoint>/processes, or ps inside a container). If the app is expected to be PID 1 in a container, still verify before collecting.
  3. Collection command: The exact command to run, including relevant providers, output format, and duration.
  4. Container considerations:
    • Collecting from inside the container: ensure the tool is installed in the image or use kubectl cp to copy it in.
    • Collecting from outside the container: use dotnet-monitor as a sidecar with a shared diagnostic port (Unix domain socket in /tmp).
    • Kubernetes: dotnet-monitor as a sidecar container, or kubectl debug for ephemeral debug containers.
  5. Long-running repros (Windows/PerfView): show how to use trigger arguments and circular buffer settings.
  6. Output location: Where the collected file will be saved and how to copy it off the target for analysis.
  7. Artifact handoff checklist: Include runtime version, OS/kernel, container image tag or build SHA, PID/process name, UTC collection start/end timestamps, exact command used, and final artifact path when handing traces to someone else for analysis.

Step 4: Recommend analysis approach

After data is collected, recommend the appropriate tool for analysis. Do not perform the analysis — just point the developer to the right tool and documentation.

Collected Data Analysis Tool Notes
.nettrace file PerfView (Windows), Speedscope (web) PerfView gives the richest view on Windows
.etl / .etl.zip file PerfView ETW traces from PerfView or perfcollect
perf.data.nl from perfcollect PerfView (Windows) Copy the file to a Windows machine and open with PerfView

Validation

  • The recommended tool is compatible with the developer's runtime, OS, and deployment topology
  • The collection command runs without errors
  • The output file is generated in the expected location
  • The developer knows which analysis tool to use for the collected data

Common Pitfalls

Pitfall Solution
Using dotnet-trace on .NET Framework dotnet-trace only works with modern .NET (.NET Core 3.0+). Use PerfView for .NET Framework.
PerfView without admin privileges PerfView requires admin for ETW tracing. Fall back to dotnet-trace if admin is not available.
perfcollect in container without SYS_ADMIN Containers drop SYS_ADMIN by default. Run with --privileged or add SYS_ADMIN capability, or fall back to dotnet-trace.
Huge trace files from long repros On Windows, use PerfView /StopOn triggers that fire on the symptom you want to capture (e.g., /StopOnPerfCounter, /StopOnGCEvent, /StopOnException) with /CircularMB and /BufferSizeMB. Never trigger on recovery — the circular buffer continuously overwrites old data, so the interesting behavior may be lost by the time collection stops.
Diagnostic port not accessible in container Mount /tmp as a shared volume between the app container and dotnet-monitor sidecar for the diagnostic Unix domain socket.
Forgetting to install tools in container image Add dotnet tool install to your Dockerfile, or use dotnet-monitor as a sidecar to avoid modifying the app image.
Exposing dotnet-monitor with --no-auth in production Keep auth enabled, bind to localhost, and use kubectl port-forward for access. Use --no-auth only for short-lived isolated debugging.
Collecting only CPU/thread-time trace for networking issues CPU and thread-time traces alone do not show HTTP status codes, DNS timing, or connection pool behavior. Add the networking providers (System.Net.Http, System.Net.NameResolution, System.Net.Security, System.Net.Sockets) alongside the thread-time trace.
Enabling all networking providers when only one is needed Each networking provider adds overhead. If the issue is clearly HTTP-level (5xx status codes), System.Net.Http alone may be sufficient. Add DNS, TLS, and socket providers when the root cause is unclear.
Files (skills)
  • references
    • dotnet-monitor.md 3.5 KB 
      # dotnet-monitor

**Purpose**: REST API-based diagnostics collection, designed for container and Kubernetes environments.

| Attribute | Value |
|-----------|-------|
| OS | Windows, Linux, macOS |
| Runtime | Modern .NET (.NET Core 3.1+) |
| .NET Framework | ❌ Not supported |
| Admin required | No |
| Container | ✅ Designed for containers/K8s — runs as a sidecar |

## Installation

```bash
# As a global tool
dotnet tool install -g dotnet-monitor

# As a container image (for K8s sidecar)
# mcr.microsoft.com/dotnet/monitor
```

## Common Commands

```bash
# Production-safe default: keep auth enabled (default) and bind to loopback
dotnet-monitor collect --urls http://127.0.0.1:52323 --metricUrls http://127.0.0.1:52325

# Dev-only shortcut in isolated environments (avoid in production)
dotnet-monitor collect --urls http://127.0.0.1:52323 --no-auth
```

## Security Guidance (Production)

- Do **not** run `dotnet-monitor` with `--no-auth` on production workloads.
- Bind to localhost (`127.0.0.1`) and access through `kubectl port-forward` (or another local-only tunnel).
- If remote access is required, keep auth enabled and restrict network exposure to trusted operators.

## REST API Endpoints

```bash
# Access via local tunnel (recommended in Kubernetes)
kubectl port-forward pod/<pod-name> 52323:52323

# List processes (auth enabled by default)
curl -H "Authorization: Bearer <monitor-token>" http://localhost:52323/processes

# Collect a trace
curl -H "Authorization: Bearer <monitor-token>" -o trace.nettrace http://localhost:52323/trace?pid=<PID>&durationSeconds=30

# Collect GC trace
curl -H "Authorization: Bearer <monitor-token>" -o trace.nettrace "http://localhost:52323/trace?pid=<PID>&profile=gc-verbose&durationSeconds=30"

# Collect trace with networking providers (HTTP status codes, DNS, TLS, sockets)
curl -H "Authorization: Bearer <monitor-token>" -o trace.nettrace "http://localhost:52323/trace?pid=<PID>&durationSeconds=30&providers=System.Net.Http,System.Net.NameResolution,System.Net.Security,System.Net.Sockets"

# Get live metrics
curl -H "Authorization: Bearer <monitor-token>" http://localhost:52323/livemetrics?pid=<PID>
```

## Kubernetes Sidecar Setup

```yaml
# Pod spec with dotnet-monitor as sidecar
spec:
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - name: diag
      mountPath: /tmp
  - name: monitor
    image: mcr.microsoft.com/dotnet/monitor:latest
    args: ["collect", "--urls", "http://127.0.0.1:52323", "--metricUrls", "http://127.0.0.1:52325"]
    volumeMounts:
    - name: diag
      mountPath: /tmp
    ports:
    - containerPort: 52323
  volumes:
  - name: diag
    emptyDir: {}
```

The shared `/tmp` volume allows `dotnet-monitor` to access the app's diagnostic Unix domain socket.
Keep auth enabled (default), and access the API via `kubectl port-forward` rather than exposing a public Service unless you have explicit authentication and network controls in place.

## Trade-offs

- ✅ Purpose-built for containers and Kubernetes
- ✅ No tools needed inside the app container
- ✅ REST API is easy to automate and integrate
- ❌ Requires sidecar setup and shared volume
- ❌ Additional resource overhead from sidecar container
- ❌ Requires authentication setup (API key or `--no-auth` flag)
- ⚠️ **When running in the workload context** (console access to the container), prefer console-based tools (`dotnet-trace`, `dotnet-trace collect-linux`) to avoid authentication setup. Use `dotnet-monitor` when console access is not available or when it is already deployed.
    • dotnet-trace-collect-linux.md 3.3 KB 
      # dotnet-trace collect-linux (.NET 10+)

**Purpose**: Collects diagnostic traces using `perf_events`, a Linux OS technology. Provides native call stacks, kernel events, and machine-wide tracing that standard `dotnet-trace collect` cannot.

| Attribute | Value |
|-----------|-------|
| OS | Linux only (kernel >= 6.4 with `CONFIG_USER_EVENTS=y`) |
| Runtime | .NET 10+ only |
| .NET Framework | ❌ Not supported |
| Admin required | Yes (root) |
| Container | Works inside container (requires root) |
| glibc | >= 2.35 (not supported on Alpine 3.22, CentOS Stream 9, or RHEL 9-based distros) |

## Key Differences from `dotnet-trace collect`

| Feature | `collect` | `collect-linux` |
|---------|-----------|-----------------|
| Trace all processes simultaneously | No | Yes (default — no PID required) |
| Capture native library and kernel events | No | Yes |
| Event callstacks include native frames | No | Yes |
| Requires admin/root | No | Yes |

## Usage

```bash
# Machine-wide trace (all processes, no PID needed)
sudo dotnet-trace collect-linux

# Trace a specific process
sudo dotnet-trace collect-linux -p <PID>

# Trace a process by name
sudo dotnet-trace collect-linux -n <process-name>

# Trace for a specific duration
sudo dotnet-trace collect-linux --duration 00:00:30

# Trace with specific providers
sudo dotnet-trace collect-linux --providers Microsoft-Windows-DotNETRuntime

# Trace with networking providers (HTTP status codes, DNS, TLS, sockets)
sudo dotnet-trace collect-linux --providers System.Net.Http,System.Net.NameResolution,System.Net.Security,System.Net.Sockets

# Trace with networking providers and thread-time profile
sudo dotnet-trace collect-linux --profile thread-time --providers System.Net.Http,System.Net.NameResolution,System.Net.Security,System.Net.Sockets

# Trace with specific profiles
sudo dotnet-trace collect-linux --profile gc-verbose
sudo dotnet-trace collect-linux --profile thread-time

# Trace with additional Linux perf events
sudo dotnet-trace collect-linux --perf-events <list-of-perf-events>

# Output to a specific file
sudo dotnet-trace collect-linux -o /tmp/trace.nettrace
```

When `--providers`, `--profile`, `--clrevents`, and `--perf-events` are not specified, `collect-linux` enables these profiles by default: `dotnet-common` (lightweight .NET runtime diagnostics) and `cpu-sampling` (kernel CPU sampling).

## Container Usage

```bash
# Inside container (machine-wide — captures all processes in the container)
sudo dotnet-trace collect-linux -o /tmp/trace.nettrace

# Inside container (specific process)
sudo dotnet-trace collect-linux -p 1 -o /tmp/trace.nettrace

# Copy trace out
kubectl cp <pod>:/tmp/trace.nettrace ./trace.nettrace
```

## Trade-offs

- ✅ Machine-wide tracing without specifying a PID
- ✅ Native (unmanaged) call stacks — essential for diagnosing native interop or runtime issues
- ✅ Captures kernel events and native library events
- ✅ Designed for Linux-first workflows
- ❌ Requires root privileges
- ❌ Only available on .NET 10 and later
- ❌ Linux only (kernel >= 6.4)
- ❌ Requires glibc >= 2.35 (not all supported Linux distros qualify)
- ❌ Native code symbols must be present on disk next to the binary (or in a standard location) during trace capture for symbol resolution
- ❌ Unable to trace cross-namespace processes
    • dotnet-trace-collect.md 2.6 KB 
      # dotnet-trace collect

**Purpose**: Collect EventPipe traces for CPU profiling, event analysis, and runtime diagnostics.

| Attribute | Value |
|-----------|-------|
| OS | Windows, Linux, macOS |
| Runtime | Modern .NET (.NET Core 3.0+) |
| .NET Framework | ❌ Not supported |
| Admin required | No |
| Container | Works inside container; use `--diagnostic-port` for sidecar |

## Installation

```bash
# As a .NET global tool (requires .NET SDK)
dotnet tool install -g dotnet-trace

# Direct download via aka.ms (no SDK required — useful in containers)
# Linux x64
curl -JL https://aka.ms/dotnet-trace/linux-x64 -o dotnet-trace
chmod +x dotnet-trace

# Linux Arm64
curl -JL https://aka.ms/dotnet-trace/linux-arm64 -o dotnet-trace
chmod +x dotnet-trace

# Linux musl x64 (Alpine)
curl -JL https://aka.ms/dotnet-trace/linux-musl-x64 -o dotnet-trace
chmod +x dotnet-trace

# Windows x64
curl -JL https://aka.ms/dotnet-trace/win-x64 -o dotnet-trace.exe
```

## Common Commands

```bash
# List running .NET processes
dotnet-trace ps

# Collect a trace with the default profiles (dotnet-common and dotnet-sampled-thread-time)
dotnet-trace collect -p <PID>

# Collect with a specific profile
dotnet-trace collect -p <PID> --profile dotnet-sampled-thread-time
dotnet-trace collect -p <PID> --profile gc-verbose

# Collect with specific providers
dotnet-trace collect -p <PID> --providers Microsoft-DotNETCore-SampleProfiler,Microsoft-Windows-DotNETRuntime

# Collect with networking providers (HTTP status codes, DNS, TLS, sockets)
dotnet-trace collect -p <PID> --providers System.Net.Http,System.Net.NameResolution,System.Net.Security,System.Net.Sockets

# Collect for a fixed duration (time span in hh:mm:ss format)
dotnet-trace collect -p <PID> --duration 00:00:30

# Output in Speedscope format for web-based viewing
dotnet-trace collect -p <PID> --format speedscope
```

## Output Formats

| Format | Extension | Analysis Tool |
|--------|-----------|---------------|
| `nettrace` (default) | `.nettrace` | PerfView, Visual Studio, `dotnet-trace report` |
| `speedscope` | `.speedscope.json` | Speedscope |
| `chromium` | `.chromium.json` | Chrome DevTools (chrome://tracing) |

## Container Usage

```bash
# Inside container (process is typically PID 1)
dotnet-trace collect -p 1 -o /tmp/trace.nettrace

# Copy trace out of container
kubectl cp <pod>:/tmp/trace.nettrace ./trace.nettrace
# or
docker cp <container>:/tmp/trace.nettrace ./trace.nettrace
```

## Trade-offs

- ✅ Cross-platform, no admin needed, lightweight
- ❌ No native (unmanaged) call stacks — only managed frames
- ❌ Less system-level detail than ETW (PerfView) or perf (perfcollect)
    • perfcollect.md 4.3 KB 
      # perfcollect

**Purpose**: Wrapper script for `perf` and `LTTng` on Linux. Collects CPU profiles with native call stacks.

| Attribute | Value |
|-----------|-------|
| OS | Linux only |
| Runtime | Modern .NET (.NET Core 2.0+) |
| .NET Framework | ❌ Not supported |
| Admin required | Yes (root) |
| Container | Needs `SYS_ADMIN` / `--privileged` |

## Installation

```bash
# Download the script
curl -OL https://aka.ms/perfcollect
chmod +x perfcollect

# Install prerequisites (perf, LTTng)
sudo ./perfcollect install
```

## Common Commands

```bash
# Collect a trace (runs until Ctrl+C)
sudo ./perfcollect collect mytrace

# Collect for a specific duration
sudo timeout 30 ./perfcollect collect mytrace
```

## Container Usage

```bash
# Docker — run with privileged mode
docker run --privileged ...
```

## Kubernetes Usage

In Kubernetes, use a **diagnostics sidecar container** to avoid modifying your application image. The sidecar runs alongside your app, shares its process namespace and `/tmp` directory, and contains the diagnostic tools.

### Step 1: Add a privileged diagnostics sidecar

Add a diagnostics container to your deployment and mark it as privileged:

```yaml
      - name: diagnostics-container
        image: ubuntu
        command: ["/bin/sh", "-c", "sleep infinity"]
        securityContext:
          privileged: true
          allowPrivilegeEscalation: true
        volumeMounts:
        - name: shared-tmp
          mountPath: /tmp
```

### Step 2: Enable shared process namespace

Set `shareProcessNamespace: true` in the pod spec so the sidecar can see and profile the app's processes:

```yaml
    spec:
      shareProcessNamespace: true
```

### Step 3: Share /tmp between containers

Create an `emptyDir` volume mounted at `/tmp` in both the app and sidecar containers. This is needed for perf map files (`perf-$pid.map`), perfcollect logs, and other profiling artifacts:

```yaml
      volumes:
      - name: shared-tmp
        emptyDir: {}
```

Mount it in both containers:

```yaml
        volumeMounts:
        - name: shared-tmp
          mountPath: /tmp
```

### Step 4: Set environment variables in the app container

Enable perf map generation and LTTng event logging in the application container:

```yaml
        env:
        - name: COMPlus_PerfMapEnabled
          value: "1"
        - name: COMPlus_EnableEventLog
          value: "1"
```

### Step 5: Collect the trace

Connect to the cluster node, exec into the diagnostics sidecar, install perfcollect, and capture:

```bash
# Exec into the diagnostics sidecar
kubectl exec -it <pod-name> -c diagnostics-container -- /bin/bash

# Inside the sidecar: install perfcollect
curl -OL https://aka.ms/perfcollect
chmod +x perfcollect
./perfcollect install

# Capture a trace
./perfcollect collect mytrace
```

This produces `mytrace.trace.zip`, which can be copied out and analyzed.

### End-to-end Kubernetes deployment example

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample
  template:
    metadata:
      labels:
        app: sample
    spec:
      shareProcessNamespace: true
      volumes:
      - name: shared-tmp
        emptyDir: {}
      containers:
      - name: sample-app
        image: myapp:latest
        ports:
        - containerPort: 80
        env:
        - name: COMPlus_PerfMapEnabled
          value: "1"
        - name: COMPlus_EnableEventLog
          value: "1"
        volumeMounts:
        - name: shared-tmp
          mountPath: /tmp
      - name: diagnostics-container
        image: ubuntu
        command: ["/bin/sh", "-c", "sleep infinity"]
        securityContext:
          privileged: true
          allowPrivilegeEscalation: true
        volumeMounts:
        - name: shared-tmp
          mountPath: /tmp
```

## Analyzing perfcollect Output

The output is a `*.trace.zip` file containing `perf.data.nl`. To analyze:

1. Copy the `.trace.zip` to a Windows machine
2. Open with PerfView — it can read perfcollect output and display flame graphs

## Trade-offs

- ✅ Captures native (unmanaged) call stacks — essential for diagnosing native interop or runtime issues
- ✅ Uses kernel-level `perf` for accurate CPU profiling
- ❌ Requires root/admin privileges
- ❌ Linux only
- ❌ In containers, requires `SYS_ADMIN` or `--privileged`
    • perfview.md 7.4 KB 
      # PerfView

**Purpose**: ETW-based tracing on Windows. The richest diagnostic tool for .NET on Windows.

| Attribute | Value |
|-----------|-------|
| OS | Windows only |
| Runtime | .NET Framework ✅, Modern .NET ✅ |
| Admin required | Yes |
| Container | ✅ Windows containers (Hyper-V and process-isolation) |

## Installation

Download from [https://github.com/microsoft/perfview/releases](https://github.com/microsoft/perfview/releases). PerfView is a standalone `.exe` — no installation required.

## Common Commands

Always include `/BufferSizeMB:1024 /CircularMB:2048` for short traces to ensure adequate buffer space.

```powershell
# Collect a CPU trace (default providers)
PerfView collect /BufferSizeMB:1024 /CircularMB:2048

# Collect for slow request / latency investigation (ThreadTime adds thread-level wait/block detail)
PerfView /ThreadTime collect /BufferSizeMB:1024 /CircularMB:2048

# Collect with GC and allocation events
PerfView collect /GCCollectOnly /BufferSizeMB:1024 /CircularMB:2048

# Collect with specific providers
PerfView collect /Providers:Microsoft-Windows-DotNETRuntime /BufferSizeMB:1024 /CircularMB:2048

# Collect with networking providers (HTTP status codes, DNS, TLS, sockets)
PerfView /ThreadTime collect /BufferSizeMB:1024 /CircularMB:2048 /Providers:*System.Net.Http,*System.Net.NameResolution,*System.Net.Security,*System.Net.Sockets

# Collect with Kestrel/ASP.NET providers for slow inbound requests
PerfView /ThreadTime collect /BufferSizeMB:1024 /CircularMB:2048 /Providers:*Microsoft.AspNetCore.Hosting,*Microsoft-AspNetCore-Server-Kestrel

# Collect for a specific duration (seconds)
PerfView collect /MaxCollectSec:30 /BufferSizeMB:1024 /CircularMB:2048

# Collect from command line without GUI
PerfView /nogui collect /MaxCollectSec:30 /BufferSizeMB:1024 /CircularMB:2048
```

## Trigger Arguments for Long-Running Repros

When the issue takes a long time to reproduce, use trigger arguments with a circular buffer to capture the issue without generating huge trace files.

**Important**: The `/StopOn` trigger should fire on the **symptom you want to capture** — not on the recovery. PerfView uses a circular buffer (`/CircularMB`) that continuously overwrites old data, so the most recent data before the stop trigger fires is what gets preserved. When a `/StopOn` trigger fires, PerfView checks the condition a few times over several seconds before actually stopping, so the trigger event and surrounding context are reliably captured.

Use `/StartOn` only when you know the start event happens **before** the stop event (e.g., to avoid recording idle time before the issue begins). If in doubt, omit `/StartOn` and just use `/StopOn` with a circular buffer.

**Note**: For **slow requests**, do not include a stop trigger by default — the right trigger depends on the specific scenario. Provide the collection command without a trigger and let the user design one based on what they're seeing.

```powershell
# Stop when CPU spikes — captures the high-CPU window
PerfView collect /StopOnPerfCounter:"Processor:% Processor Time:_Total>80" /BufferSizeMB:2048 /CircularMB:4096

# Stop on a GC event (e.g. Gen2 collection)
PerfView collect /StopOnGCEvent /BufferSizeMB:2048 /CircularMB:4096

# Stop when a specific exception is thrown (not recommended for memory leaks — capture during growth instead)
PerfView collect /StopOnException:"System.OutOfMemoryException" /BufferSizeMB:2048 /CircularMB:4096

# StartOn + StopOn — only when the start event is known to precede the stop event
# Here: start recording when CPU goes above 50%, stop when the spike hits 90%
PerfView collect /StartOnPerfCounter:"Processor:% Processor Time:_Total>50" /StopOnPerfCounter:"Processor:% Processor Time:_Total>90" /BufferSizeMB:2048 /CircularMB:4096
```

Key trigger parameters:

| Parameter | Description |
|-----------|-------------|
| `/StopOnPerfCounter:"Category:Counter:Instance>Threshold"` | Stop when a performance counter crosses a threshold. **Use this on the symptom you want to capture.** |
| `/StartOnPerfCounter:"Category:Counter:Instance>Threshold"` | Start collection when a counter crosses a threshold. Only use when you know this fires before the stop trigger. |
| `/StopOnGCEvent` | Stop when a GC event occurs |
| `/StopOnException:"ExceptionType"` | Stop when a specific exception is thrown |
| `/BufferSizeMB:N` | In-memory buffer size (increase for long collections) |
| `/CircularMB:N` | Circular log size on disk (keeps only the last N MB of data) |

## Windows Container Usage

PerfView works with both types of Windows containers. Most Windows containers (including Kubernetes on Windows) use **process-isolation** by default.

> **PerfView inside containers**: `PerfView.exe` does not run in many slimmed-down Windows container images. To run PerfView inside a container (needed for the merge step in process-isolation, or for direct collection in Hyper-V), build **PerfViewCollect** from [https://github.com/microsoft/perfview](https://github.com/microsoft/perfview) as a self-contained publish, then copy the output binaries into the container.

### Process-isolation containers (default)

Process-isolation containers share the host kernel. Collect from **outside** the container (on the host) using `/EnableEventsInContainers`:

```powershell
# On the host — captures all processes including those inside containers
PerfView collect /EnableEventsInContainers /MaxCollectSec:30 /BufferSizeMB:1024 /CircularMB:2048
```

The resulting trace can be analyzed immediately on the host while the container is still running — PerfView can reach into the container to fetch binaries for symbol resolution.

**To analyze on another machine**, you must complete a merge step inside the container **before the container shuts down**. Without this, binaries inside the container won't have their symbol lookup information saved in the trace:

```powershell
# 1. Copy the .etl.zip into the container
docker cp trace.etl.zip <container>:C:\trace.etl.zip

# 2. Inside the container — complete the merge to embed symbol info
PerfViewCollect merge /ImageIDsOnly C:\trace.etl.zip

# 3. Copy the merged trace back out
docker cp <container>:C:\trace.etl.zip ./trace-merged.etl.zip
```

The merged trace can now be analyzed on any machine with full symbol resolution. If you skip the in-container merge step, symbols for binaries that live only inside the container will be unresolvable on other machines because the Windows merge component cannot reach into the container's filesystem.

### Hyper-V containers

Hyper-V containers are less common and are effectively lightweight VMs. Collect traces from **inside** the container the same way you would on a regular machine:

```powershell
# Inside the Hyper-V container
PerfView collect /MaxCollectSec:30 /BufferSizeMB:1024 /CircularMB:2048
```

## Trade-offs

- ✅ Richest diagnostic data available on Windows (ETW kernel + CLR providers)
- ✅ Works with both .NET Framework and modern .NET
- ✅ Powerful trigger system for capturing hard-to-reproduce issues
- ✅ Works with Windows containers (Hyper-V and process-isolation)
- ❌ Windows only
- ❌ Requires admin privileges
- ⚠️ **For .NET Framework without admin**: PerfView is the only trace tool for .NET Framework, and it requires admin. Without admin, there is **no way to investigate high CPU, slow requests, or excessive GC** on .NET Framework. Dumps can help with hangs and memory leaks (delegate to the `dump-collect` skill), but trace-based investigation requires admin access.
  • SKILL.md 24.1 KB 
    ---
name: dotnet-trace-collect
description: Guide developers through capturing diagnostic artifacts to diagnose production .NET performance issues. Use when the user needs help choosing diagnostic tools, collecting performance data, or understanding tool trade-offs across different environments (Windows/Linux, .NET Framework/modern .NET, container/non-container).
license: MIT
---

# .NET Trace Collect

This skill helps developers diagnose production performance issues by recommending the right diagnostic tools for their environment, guiding data collection, and suggesting analysis approaches. It does not analyze code for anti-patterns or perform the analysis itself.

## When to Use

- A developer needs to investigate a production performance issue (high CPU, memory leak, slow requests, excessive GC, networking errors, etc.)
- Choosing the right diagnostic tool for a specific runtime, OS, or deployment topology
- Setting up and running diagnostic tool commands for data collection
- Understanding trade-offs between available tools (e.g. PerfView vs dotnet-trace)
- Collecting diagnostics from containerized or Kubernetes workloads

## When Not to Use

- Reviewing source code for performance anti-patterns (use a code review skill instead)
- Benchmarking during development (e.g. BenchmarkDotNet setup)
- Analyzing collected trace or dump files (this skill recommends tools for analysis, but does not perform it)

## Inputs

| Input | Required | Description |
|-------|----------|-------------|
| Symptom | Yes | What the developer is observing (high CPU, memory growth, slow requests, hangs, excessive GC, HTTP 5xx errors, networking timeouts, connection failures, assembly loading failures, etc.) |
| Runtime | Yes | .NET Framework or modern .NET (and version, especially whether .NET 10+) |
| OS | Yes | Windows or Linux |
| Deployment | Yes | Non-container, container, or Kubernetes |
| Admin privileges | Recommended | Whether the developer has admin/root access on the target machine |
| Repro characteristics | Recommended | Whether the issue is easy to reproduce or requires a long time to manifest |

## Workflow

### Step 1: Understand the environment

Determine or ask the developer to clarify:

1. **Symptom**: What they are observing (high CPU, memory leak, slow requests, hangs, excessive GC, HTTP 5xx errors, networking timeouts, connection failures, assembly loading failures, etc.)
2. **Runtime**: .NET Framework or modern .NET? If modern .NET, which version? (Especially whether .NET 10 or later.)
3. **OS**: Windows or Linux?
4. **Deployment**: Running directly on the host, in a container, or in Kubernetes?
5. **Admin privileges**: Do they have admin/root access on the target machine or container?
6. **Repro characteristics**: Does the issue reproduce quickly, or does it take a long time to manifest?
7. **Workload context**: Determine or ask the user if you are running in the context of the workload (i.e., on the same machine or connected to the same environment where the issue is occurring). If so, you can run diagnostic commands directly on their behalf. If not, provide the commands as guidance for the user to run themselves.

Use this information to select the right tool in Step 2.

### Step 2: Recommend diagnostic tools

Select tools based on the environment using the priority rules below. Once a tool is selected, load the corresponding reference file for detailed command-line usage.

#### Tool reference lookup

| Environment | Reference file(s) |
|-------------|-------------------|
| Windows + modern .NET + admin | `references/perfview.md` |
| Windows + modern .NET, no admin | `references/dotnet-trace-collect.md` |
| Windows + .NET Framework | `references/perfview.md` |
| Linux + .NET 10+ + root | `references/dotnet-trace-collect-linux.md` |
| Linux + pre-.NET 10 | `references/dotnet-trace-collect.md` |
| Linux + native stacks needed | `references/perfcollect.md` |
| Container/K8s (console access) | `references/dotnet-trace-collect.md` (or `dotnet-trace-collect-linux.md`) |
| Container/K8s (no console) | `references/dotnet-monitor.md` |

#### Quick decision matrix (first-pass triage)

| Environment | Preferred tool | Fallback / Notes |
|-------------|----------------|------------------|
| Windows + modern .NET + admin | PerfView | If admin is unavailable, use `dotnet-trace` |
| Windows + .NET Framework + admin | PerfView | Without admin, there is no trace fallback; for hangs/memory leaks, provide dump commands directly (`procdump -ma` or Task Manager) since `dump-collect` does not support .NET Framework |
| Linux + .NET 10+ + root | `dotnet-trace collect-linux` | Use `dotnet-trace` if root or kernel prerequisites are not met |
| Linux + pre-.NET 10 | `dotnet-trace` | Add `perfcollect` when native stacks are needed (requires root) |
| Linux container/Kubernetes | Console tools if in workload context; `dotnet-monitor` if no console access | See Linux Container / Kubernetes section for details |

#### Windows (non-container, modern .NET)

1. **PerfView** (preferred) — produces richer ETW-based data; requires admin privileges. For **slow requests**, add `/ThreadTime` to capture thread-level wait and block detail.
2. **`dotnet-trace`** — fallback when admin privileges are not available.
3. For **long-running repros**: use PerfView with a `/StopOn` trigger that fires on the **symptom you want to capture** (e.g., `/StopOnPerfCounter`, `/StopOnGCEvent`, `/StopOnException`) and a circular buffer (`/CircularMB` + `/BufferSizeMB`). **Critical: the stop trigger must fire on the interesting event, not the recovery.** The circular buffer continuously overwrites old data, so if you trigger on recovery, the buffer may have already overwritten the interesting behavior by the time collection stops. Only add `/StartOn` if the start event is known to precede the stop event. For **slow requests**, do not include a stop trigger by default — let the user design one based on their specific scenario.

#### Windows containers

1. **PerfView** — most Windows containers (including Kubernetes on Windows) use process-isolation by default. Collect from the host with `/EnableEventsInContainers`. After collection, you have two options:
   - **Analyze locally while the container is still running** — PerfView can reach into the live container to resolve symbols, so you can open the trace immediately on the host machine.
   - **Analyze off-machine** — before the container shuts down, copy the `.etl.zip` into the container and run `PerfViewCollect merge /ImageIDsOnly` inside it to embed symbol information. Then copy the merged trace out. Without this merge step, symbols for binaries inside the container will be unresolvable on other machines.

   For the less common Hyper-V containers, collect inside the container directly. See [references/perfview.md](references/perfview.md) for detailed commands.
2. **`dotnet-monitor`**, **`dotnet-trace`** — inside the container if the tools are installed in the image. For dumps, invoke the **`dump-collect`** skill.

#### Windows (.NET Framework)

1. **PerfView** — the primary diagnostic tool for .NET Framework on Windows. Requires admin.
2. Same trigger guidance for long repros: use `/StopOn` triggers that fire on the symptom (e.g., `/StopOnPerfCounter`, `/StopOnGCEvent`, `/StopOnException`) with `/CircularMB` + `/BufferSizeMB`.
3. **Without admin**: PerfView requires admin, and there are no alternative trace tools for .NET Framework. Process dumps can still be captured without admin — provide dump commands directly (e.g., `procdump -ma <PID>` or Task Manager) since the `dump-collect` skill does not support .NET Framework. Dumps can help diagnose hangs and memory leaks. However, for **high CPU**, **slow requests**, and **excessive GC**, there is no way to investigate on .NET Framework without admin access. Advise the user to obtain admin privileges.

#### Linux (non-container, .NET 10+)

1. **`dotnet-trace collect-linux`** (preferred) — uses `perf_events` for richer traces including native call stacks and kernel events. Captures machine-wide by default (no PID required). Requires root and kernel >= 6.4.
2. **`dotnet-trace`** — fallback when root privileges are not available or kernel requirements are not met. Managed stacks only.

#### Linux (non-container, pre-.NET 10)

1. **`dotnet-trace`** (preferred) — managed trace collection; no admin required.
2. **`perfcollect`** — when **native call stacks** are needed (requires admin/root).

#### Linux Container / Kubernetes

**If running in the context of the workload** (i.e., you have console access to the container), prefer console-based tools. These are easier to set up than `dotnet-monitor`, which requires authentication configuration and sidecar deployment:

1. **`dotnet-trace collect-linux`** (.NET 10+ with root) — produces the richest traces including native call stacks and kernel events.
2. **`dotnet-trace`** — inside the container if the tool is installed in the image. For dumps, invoke the **`dump-collect`** skill.
3. **`perfcollect`** — inside the container when native stacks are needed on pre-.NET 10 (requires `SYS_ADMIN` / `--privileged`).

**If not running in the workload context** (no console access), or if `dotnet-monitor` is already deployed:

1. **`dotnet-monitor`** — designed for containers; runs as a sidecar. No tools needed in the app container. Easiest option when console access is not available.

#### Memory dumps

When dumps are needed (memory leaks, hangs), **do not provide dump collection commands directly** for modern .NET — invoke the **`dump-collect`** skill instead. The `dump-collect` skill only supports modern .NET (.NET Core 3.0+). For **.NET Framework**, provide dump collection guidance directly (e.g., `procdump -ma <PID>` or Task Manager). This skill focuses on trace collection only.

#### Memory leaks

- **Capture two dumps** as memory is increasing (e.g., one early, one after significant growth). Invoke the **`dump-collect`** skill for dump collection — do not provide dump commands directly. Diff the dumps in PerfView to see which objects have increased — this is the most effective way to identify what is leaking.
- **Without admin privileges**: Two process dumps can give a sense of what's growing on the heap, but may not be enough to identify the root cause. If dumps aren't sufficient, reproduce the issue in an environment where admin privileges are available to collect richer data (traces).
- **Modern .NET on Linux (pre-.NET 10)**: Recommend two dump captures (invoke `dump-collect` skill) for heap diff, plus `dotnet-trace` while memory is growing (for allocation tracking). No trigger needed — capture during the growth period. Both together give the best picture.
- **Modern .NET 10+ on Linux with admin**: Recommend two dump captures (invoke `dump-collect` skill) for heap diff, plus `dotnet-trace collect-linux` while memory is growing (richer data including native stacks). No trigger needed.
- **.NET Framework**: Recommend two dumps plus a PerfView trace while memory is growing to see what is being allocated. The `dump-collect` skill does not support .NET Framework, so provide dump commands directly (e.g., `procdump -ma <PID>` or right-click → Create Dump File in Task Manager). No trigger is needed — just capture the trace during the growth period. Do not wait for an `OutOfMemoryException`.

#### Excessive GC

Excessive GC requires a **trace** to analyze GC events, pause times, and allocation patterns — a dump is not sufficient.

- **Windows (PerfView)**: Use `PerfView collect /GCCollectOnly` to capture GC events.
- **Linux (dotnet-trace)**: Use `dotnet-trace collect -p <PID> --profile gc-verbose`.
- **Linux .NET 10+ with root**: Use `dotnet-trace collect-linux --profile gc-verbose` for richer data with native stacks.
- **Containers**: `dotnet-monitor` can capture GC traces via its REST API (`/trace?profile=gc-verbose`).

#### Slow Requests

Slow requests require a **thread time trace** to see where threads are spending time — waiting on locks, I/O, external calls, etc. Use larger buffers since thread time traces generate more data. For ASP.NET Core applications, also enable `Microsoft.AspNetCore.Hosting` and `Microsoft-AspNetCore-Server-Kestrel` providers to get server-side request lifecycle timing (when requests arrive, how long they take to process).

- **Windows (PerfView)**: Use `PerfView /ThreadTime collect /BufferSizeMB:1024 /CircularMB:2048`. The `/ThreadTime` argument adds thread-level wait and block detail. For ASP.NET Core, add Kestrel providers: `PerfView /ThreadTime collect /BufferSizeMB:1024 /CircularMB:2048 /Providers:*Microsoft.AspNetCore.Hosting,*Microsoft-AspNetCore-Server-Kestrel`. Do not include a stop trigger by default — let the user design one based on their specific scenario.
- **Linux (dotnet-trace)**: `dotnet-trace` captures thread time data by default — no special arguments needed. Use `dotnet-trace collect -p <PID>`. For ASP.NET Core, add Kestrel providers: `dotnet-trace collect -p <PID> --providers Microsoft.AspNetCore.Hosting,Microsoft-AspNetCore-Server-Kestrel`.
- **Linux .NET 10+ with root**: Use `dotnet-trace collect-linux --profile thread-time` for richer data with native stacks. For ASP.NET Core, add: `--providers Microsoft.AspNetCore.Hosting,Microsoft-AspNetCore-Server-Kestrel`.
- **Containers**: `dotnet-monitor` can capture traces via its REST API (`/trace?pid=<PID>&durationSeconds=30`).

#### Hangs

1. **Start with a trace** to understand what threads are doing. Use the appropriate trace tool for the environment (PerfView with `/ThreadTime` on Windows, `dotnet-trace` on Linux, `dotnet-trace collect-linux --profile thread-time` on .NET 10+ Linux with root). The trace can reveal:
   - **Livelocks** (threads spinning without forward progress) — threads appear busy but the application makes no progress.
   - **Thread starvation** — the ThreadPool is exhausted and queued work items are not being processed. This can look like a deadlock but has a different root cause.
   - **Whether there is any forward progress at all** — if some threads are making progress, the issue may be a bottleneck rather than a true hang.
2. **If the trace does not explain the hang**, the issue may be a **true deadlock** (threads waiting on each other in a cycle). In this case, invoke the **`dump-collect`** skill to collect a process dump — do not provide dump commands directly.
3. **Analyze the dump with a debugger** to inspect thread stacks and identify the lock cycle:
   - **Windows**: Visual Studio or WinDbg with the SOS debugger extension.
   - **Linux**: `lldb` with the SOS debugger extension.

#### Networking Issues

Networking issues (HTTP 5xx errors from downstream services, request timeouts, connection failures, DNS resolution failures, TLS handshake failures, connection pool exhaustion) require **both** a thread-time trace and networking event providers. The thread-time trace shows where threads are blocked (slow downstream calls, thread starvation), while the networking events show the request lifecycle — which requests failed, what status codes came back, how long DNS resolution and TLS handshakes took, and how long requests waited for a connection from the pool.

For **.NET Framework**, `PerfView /ThreadTime` already collects the relevant networking events (from the `System.Net` ETW provider) — no additional providers are needed.

For **modern .NET**, you must explicitly enable the `System.Net.*` EventSource providers:

| Provider | What it covers |
|----------|---------------|
| `System.Net.Http` | HttpClient/SocketsHttpHandler — request lifecycle, HTTP status codes, connection pool |
| `System.Net.NameResolution` | DNS lookups (start/stop, duration) |
| `System.Net.Security` | TLS/SSL handshakes (SslStream) |
| `System.Net.Sockets` | Low-level socket connect/disconnect |

Key events from `System.Net.Http`: `RequestStart` (scheme, host, port, path), `RequestStop` (statusCode — `-1` if no response was received), `RequestFailed` (exception message for timeouts, connection refused, etc.), `RequestLeftQueue` (time waiting for a connection from the pool — indicates connection pool exhaustion), `ConnectionEstablished`, `ConnectionClosed`.

Collect a thread-time trace with networking providers enabled (modern .NET only — .NET Framework needs only `PerfView /ThreadTime`):

- **Windows (PerfView)**: Use `PerfView /ThreadTime collect /BufferSizeMB:1024 /CircularMB:2048 /Providers:*System.Net.Http,*System.Net.NameResolution,*System.Net.Security,*System.Net.Sockets`. For .NET Framework, omit the `/Providers` flag — `/ThreadTime` already includes the networking events. The thread-time trace shows where threads are blocked while the networking events show what requests are failing and why.
- **Linux (dotnet-trace)**: `dotnet-trace` captures thread time data by default, but specifying `--providers` overrides the defaults so you must also include `--profile`: `dotnet-trace collect -p <PID> --profile dotnet-common,dotnet-sampled-thread-time --providers System.Net.Http,System.Net.NameResolution,System.Net.Security,System.Net.Sockets`.
- **Linux .NET 10+ with root**: Use `dotnet-trace collect-linux --profile dotnet-common,cpu-sampling,thread-time --providers System.Net.Http,System.Net.NameResolution,System.Net.Security,System.Net.Sockets`.
- **Containers**: `dotnet-monitor` can capture traces with custom providers via its REST API.

#### Assembly Loading Issues

For modern .NET, assembly loading issues (`FileNotFoundException`, `FileLoadException`, `ReflectionTypeLoadException`, version conflicts, duplicate assembly loads across AssemblyLoadContexts) require collecting **assembly loader binder events** from the `Microsoft-Windows-DotNETRuntime` provider with the Loader keyword (`0x4`). These events trace every step of the runtime's assembly resolution algorithm — which paths were probed, which AssemblyLoadContext handled the load, whether the load succeeded or failed, and why. For .NET Framework, the same provider and keyword work for ETW-based collection; additionally, the Fusion Log Viewer (`fuslogvw.exe`) can diagnose assembly binding failures without requiring a trace.

The provider specification is `Microsoft-Windows-DotNETRuntime:0x4:4` (provider name, AssemblyLoader keyword, Informational verbosity).

- **Windows (PerfView)**: A default PerfView trace already includes binder events - simply run `PerfView collect` with no extra providers. For a smaller trace file, use `PerfView collect /ClrEvents:Default-Profile`, which removes the most verbose default events while keeping the events necessary for diagnosing assembly loading issues.
- **Linux / cross-platform (dotnet-trace)**: Use `dotnet-trace collect --clrevents assemblyloader -- <path-to-built-exe>` to launch and trace the process, or `dotnet-trace collect --clrevents assemblyloader -p <PID>` to attach to a running process.
- **Linux .NET 10+ with root**: Use `dotnet-trace collect-linux --clrevents assemblyloader`.
- **Containers**: `dotnet-monitor` can capture traces with the loader provider via its REST API.

For short-lived processes that fail on startup (common with assembly loading issues), prefer the `dotnet-trace` launch form (`-- <path-to-built-exe>`) over attaching by PID, since the process may exit before you can attach.

Explain the trade-offs when recommending a tool. For example:
- PerfView gives richer data but needs admin; runs on Windows including Windows containers.
- `dotnet-trace` works cross-platform without admin but captures less system-level detail.
- `perfcollect` captures native call stacks but needs admin/root.
- `dotnet-monitor` is the best option for containers/K8s when console access is not available, but requires sidecar deployment and authentication configuration.

### Step 3: Guide data collection

Provide the specific commands for the recommended tool. Load the appropriate reference file from the [tool reference lookup](#tool-reference-lookup) table for detailed command-line examples.

Key guidance to include:

1. **Installation**: How to install the tool if it is not already available (e.g. `dotnet tool install -g dotnet-trace`). **When recommending multiple tools, provide installation and usage instructions for each one** — do not mention a tool without showing how to install and use it.
2. **PID discovery (required before any `-p <PID>` command)**: Verify the target process first (for example: `dotnet-trace ps`, `curl <monitor-endpoint>/processes`, or `ps` inside a container). If the app is expected to be PID 1 in a container, still verify before collecting.
3. **Collection command**: The exact command to run, including relevant providers, output format, and duration.
4. **Container considerations**:
   - Collecting from **inside** the container: ensure the tool is installed in the image or use `kubectl cp` to copy it in.
   - Collecting from **outside** the container: use `dotnet-monitor` as a sidecar with a shared diagnostic port (Unix domain socket in `/tmp`).
   - Kubernetes: `dotnet-monitor` as a sidecar container, or `kubectl debug` for ephemeral debug containers.
5. **Long-running repros** (Windows/PerfView): show how to use trigger arguments and circular buffer settings.
6. **Output location**: Where the collected file will be saved and how to copy it off the target for analysis.
7. **Artifact handoff checklist**: Include runtime version, OS/kernel, container image tag or build SHA, PID/process name, UTC collection start/end timestamps, exact command used, and final artifact path when handing traces to someone else for analysis.

### Step 4: Recommend analysis approach

After data is collected, recommend the appropriate tool for analysis. Do **not** perform the analysis — just point the developer to the right tool and documentation.

| Collected Data | Analysis Tool | Notes |
|----------------|---------------|-------|
| `.nettrace` file | PerfView (Windows), Speedscope (web) | PerfView gives the richest view on Windows |
| `.etl` / `.etl.zip` file | PerfView | ETW traces from PerfView or perfcollect |
| `perf.data.nl` from perfcollect | PerfView (Windows) | Copy the file to a Windows machine and open with PerfView |

## Validation

- [ ] The recommended tool is compatible with the developer's runtime, OS, and deployment topology
- [ ] The collection command runs without errors
- [ ] The output file is generated in the expected location
- [ ] The developer knows which analysis tool to use for the collected data

## Common Pitfalls

| Pitfall | Solution |
|---------|----------|
| Using `dotnet-trace` on .NET Framework | `dotnet-trace` only works with modern .NET (.NET Core 3.0+). Use PerfView for .NET Framework. |
| PerfView without admin privileges | PerfView requires admin for ETW tracing. Fall back to `dotnet-trace` if admin is not available. |
| `perfcollect` in container without `SYS_ADMIN` | Containers drop `SYS_ADMIN` by default. Run with `--privileged` or add `SYS_ADMIN` capability, or fall back to `dotnet-trace`. |
| Huge trace files from long repros | On Windows, use PerfView `/StopOn` triggers that fire on the symptom you want to capture (e.g., `/StopOnPerfCounter`, `/StopOnGCEvent`, `/StopOnException`) with `/CircularMB` and `/BufferSizeMB`. **Never trigger on recovery** — the circular buffer continuously overwrites old data, so the interesting behavior may be lost by the time collection stops. |
| Diagnostic port not accessible in container | Mount `/tmp` as a shared volume between the app container and `dotnet-monitor` sidecar for the diagnostic Unix domain socket. |
| Forgetting to install tools in container image | Add `dotnet tool install` to your Dockerfile, or use `dotnet-monitor` as a sidecar to avoid modifying the app image. |
| Exposing `dotnet-monitor` with `--no-auth` in production | Keep auth enabled, bind to localhost, and use `kubectl port-forward` for access. Use `--no-auth` only for short-lived isolated debugging. |
| Collecting only CPU/thread-time trace for networking issues | CPU and thread-time traces alone do not show HTTP status codes, DNS timing, or connection pool behavior. Add the networking providers (`System.Net.Http`, `System.Net.NameResolution`, `System.Net.Security`, `System.Net.Sockets`) alongside the thread-time trace. |
| Enabling all networking providers when only one is needed | Each networking provider adds overhead. If the issue is clearly HTTP-level (5xx status codes), `System.Net.Http` alone may be sufficient. Add DNS, TLS, and socket providers when the root cause is unclear. |

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.

Related

Claude Prompt
Tighten-my-prose editing prompt
Claude Skill
Adversarial code-review skill for Claude
Claude Workflow
Deep-research workflow (fan-out + verify)
Claude Skill
Conventional-commit writer for Claude