Context:

High CPU consumption in IIS worker processes (w3wp.exe) can significantly impact the performance of your web applications. Here we will discuss available tools to identifying symptoms, initial troubleshooting steps and data collection methods.

Symptoms:

When IIS worker processes consume high CPU, you may notice:

Slow response times or timeouts in web applications.
High memory usage accompanying the high CPU.
Overall server performance degradation.

Initial isolation:

Troubleshooting high CPU involves two main tasks: Issue identification, Log Collection and Analysis. Here are some isolation to help you get started with IIS worker process:

Is high CPU accompanied by high memory usage or slowness? Observe the memory consumption along with CPU consumption.
Describe the steps leading to high CPU usage. Dose the CPU consumptions increase with load?
What is the CPU consumption by w3wp.exe alone and the total CPU consumption on the server during the issue? If overall CPU is high, then there may be an issue with server performance.
How does CPU consumption change over time? If this spikes up and down then should be as per the code execution. Monitor and if stays for 5-10 seconds, then we have a scenario to troubleshoot further.
How often does the problem occur and how high does the CPU get? Observe any pattern or specific time. May be any scheduled task is executing or load increases.

Data Collection

For .NET Core applications hosted on IIS, the data collection method depends on whether the app is in-process or out-of-process. In-process mode is recommended over out-of-proc.

Caution 1: For CPU analysis, either ETW trace or memory dumps can be useful. Avoid collecting both simultaneously to prevent additional CPU load or slowness.

ETW Trace vs. Memory Dumps:

ETW Trace: Useful in production scenarios with minimal performance impact. Captures specific events and performance counters.
Memory Dumps: Provides an in-depth view of objects, their values, and roots. Useful for detailed analysis.

Caution 2: Avoid collecting logs at very high CPU values (>=95%) as the server might hang or become unresponsive. Collect logs before CPU reaches that level.

The initial goal is to get logs that enable us to watch the operations on the same non-waiting thread(s) over a portion of the problematic time period where w3wp.exe CPU is highest. Therefore:

Multiple dumps are needed (3 is usually a good number).
Dumps should be taken of the same process ID.
Dumps should be close enough in time (usually 10 seconds apart).
Collect dumps within a CPU usage that is considered high and abnormal.

ETW Steps – Perfview

We will use the Perfview tool to collect traces.

Download Perfview from here.
Run PerfView.exe as admin.
During the occurrence of the issue:

Click the Collect Menu and select Collect option.
Check Zip, Merge, and Thread Time checkboxes.
Expand the Advanced Options tab and select the IIS checkbox.
Start collection and stop once the issue is produced.

Note: Do not capture this trace for too long as data gets overwritten once the circular MB limit is reached.

Memory Dumps Steps

We will use either Procdump.exe or DebugDiag Collection tool. Procdump is preferred when installation of applications is not possible.

Caution 3: Do not use Task Manager to collect dumps. Specialized tools like DebugDiag and Procdump provide more detailed information and handle process bitness correctly.

Option 1: Procdump

Download Procdump.exe from here.
In an elevated command prompt, go to the directory where you have extracted the downloaded procdump.zip and run the command:
procdump.exe -c w3wpCPUConsumptionTrigger -s NumOfSeconds -ma -n NumberOfDumpsToCollect PID

-c: The w3wp.exe CPU consumption threshold. Replace w3wpCPUConsumptionTrigger with threshold say 80%.
-s: Number of consecutive seconds the CPU consumption must be above the threshold. Replace NumOfSeconds with no of seconds CPU stays above the threshold, say 10 seconds.
-ma: Full memory dump.
-n: Number of dumps to collect. Replace NumberOfDumpsToCollect  with number of dumps to collected. Usually this is set as 3 for conclusive analysis.
PID is the process ID, you can find it from Task manager or IIS Worker Processes.

Command example : procdump.exe -c 80 -s 10 -ma -n 3 1234

Option 2: DebugDiag

Install DebugDiag from here.
Open DebugDiag Collection from the start menu.
Change the path for dumps if needed via Tools > Options And Settings > Manual Userdump Save Folder.
Click the “Processes” tab.
Locate the w3wp process by its Process ID.
Right-click the process and select “Create Userdump Series”, then set the options.
Click “Save and close” to start generating dumps.

Finally: By following these steps, you can effectively isolate and collect necessary data for high CPU consumption issues in IIS worker processes. If you are comfortable analyzing this data then you should be able to draw insights from Perfview an Dumps. If you want us to do that, please contact Microsoft IIS Support with a case and we will do it for you.

​Microsoft Tech Community – Latest Blogs –Read More