---
Title: Troubleshoot performance
URL Source: https://company-skill.com/p/alinux/alinux-troubleshoot-performance
Language: en
Description: You need to identify and fix performance bottlenecks in Alibaba Cloud Linux systems—whether caused by high CPU/sys usage, memory leaks (e.g., slab_unreclaimable), I/O stalls, network congestion, or…
---

# Troubleshoot performance

Part of **Alibaba Cloud Linux**. Route queries via `POST https://company-skill.com/api/route`.

## What You Want to Do

You need to identify and fix performance bottlenecks in Alibaba Cloud Linux systems—whether caused by high CPU/sys usage, memory leaks (e.g., `slab_unreclaimable`), I/O stalls, network congestion, or unexpected OOM kills. You may be working interactively or building automated diagnostics.

**Typical User Questions**:
- How to diagnose high CPU or memory usage?
- Why is my application experiencing scheduling delays?

## Decision Tree

Pick the best path for your situation:

- **If** you are a non-expert user needing quick automated diagnosis via GUI and have access to the **Operating System Console** with **SysOM** component ≥3.7.0 → Use **SysOM** (go to *alinux/alinux-diagnosis*)
- **If** you require deep kernel-level analysis of issues like `ksoftirqd` spikes, `io_uring` errors, or `watermark_scale_factor` tuning using tools like `bpftrace`, `ftrace`, `crash`, or `perf` → Use **** (go to *alinux/alinux-system*)
- **If** you need real-time CLI monitoring with subcommands like `sysak memleak`, `sysak nosched`, or `sysak iofsstat`, or want to automate via scripts using `sysak mservice` or analyze crashes with `kdumpctl readlog` → Use **SysAK** (go to *alinux/alinux-monitoring*)
- **Otherwise (default)** → Start with **SysOM** if you’re in Chinese mainland or Hong Kong and have proper RAM permissions; otherwise, use **SysAK** for immediate CLI-based insight without deep kernel expertise.

## Path Comparison

| Path | Best For | Complexity | Code Required | Automation | Key Fact | Detail Skill |
|------|----------|------------|---------------|------------|----------|-------------|
| CPUI/O | medium | Yes | No | Requires manual installation of `bpftrace`, `crash`, and `perf`; needs matching `kernel-debuginfo` for memory leak analysis | `alinux/troubleshooting/alinux-system` |
| SysOM | low | No | Yes | Free service; requires `AliyunSysomFullAccess` and `AliyunECSReadOnlyAccess` RAM policies | `alinux/guide/alinux-diagnosis` |
| SysAK | medium | Yes | Yes | Includes HTTP metrics endpoint at `http://127.0.0.1:9200/metrics/raw/`; `kdumpctl readlog` requires Alibaba Cloud Linux 3 kernel ≥5.10.134-14 | `alinux/cli/alinux-monitoring` |

## Path Details

### Path 1: Console / Dashboard
**Best For**: CPUI/O

**Brief Description**: This approach uses low-level command-line tools like `bpftrace`, `ftrace`, `crash`, and `perf` to perform deep kernel diagnostics. It supports analysis of `slab_unreclaimable` memory, `ksoftirqd` latency spikes, `io_uring` ENOMEM errors, and tuning of `/proc/sys/vm/watermark_scale_factor` or `/proc/async_load_calc`.

**Key technical facts**:
- Billing: N/A
- Cold start: N/A
- Max model size: N/A
- Runtimes: N/A
- Custom Docker: N/A
- Auto-scaling: N/A

**When to Use**:
- User needs to perform deep kernel-level diagnostics for issues like slab memory leaks
- Investigating high ksoftirqd latency requiring ftrace/bpftrace analysis
- Diagnosing io_uring ENOMEM errors that require ulimit adjustments
- Troubleshooting CPU sys usage spikes under memory pressure requiring watermark parameter tuning
- Analyzing eBPF LRU hash CPU spikes requiring kernel hotfix verification

**When NOT to Use**:
- User is a non-expert seeking quick automated diagnosis
- Need one-click comprehensive system health check without CLI interaction
- Looking for real-time resource monitoring dashboards instead of command-line tools
- Require historical OOM event diagnosis for destroyed instances
- Prefer GUI-based workflow over terminal commands

**Known Limitations**:
- Requires manual installation of diagnostic tools like bpftrace, crash, and perf
- Needs root or sudo privileges for most diagnostic operations
- Requires knowledge of kernel internals and debugging commands
- Some solutions require kernel hotfix installation which may need system reboot
- Memory leak analysis requires matching kernel-debuginfo package version

### Path 2: SysOM

**Brief Description**: This web-based method uses the **Operating System Console** with the **SysOM** component (≥3.7.0) to deliver **One-click Diagnosis**, **Quick Diagnostics**, **OOM Diagnosis**, and **Historical Diagnosis** for up to 14 days. It provides **Node Health** insights without CLI interaction and is ideal for users with limited kernel expertise.

**Key technical facts**:
- Billing: All system diagnosis and troubleshooting features are provided free of charge
- Regions available: Chinese mainland, China (Hong Kong)
- Auth method: RAM user must be granted AliyunECSReadOnlyAccess and AliyunSysomFullAccess system policies

**When to Use**:
- User is a non-expert seeking quick automated health checks
- Need to diagnose OOM events through web UI without CLI knowledge
- Require historical diagnosis of ACS/ECI instances even after destruction
- Looking for comprehensive one-click diagnosis with unified diagnostic report
- Prefer GUI-based workflow with form inputs and button clicks

**When NOT to Use**:
- User needs real-time command-line monitoring of system resources
- Require deep kernel-level debugging with tools like bpftrace or crash
- Working in regions outside Chinese mainland or China (Hong Kong)
- Lack required RAM permissions for SysOM access
- Need to automate diagnostics through scripts rather than manual UI interaction

**Known Limitations**:
- Only available in Chinese mainland and China (Hong Kong) regions
- Requires specific RAM permissions (AliyunECSReadOnlyAccess and AliyunSysomFullAccess)
- Historical diagnosis limited to events within the last 14 days
- Requires SysOM component version 3.7.0 or later to be installed
- Instance must be enrolled in management for node health monitoring

### Path 3: SysAK

**Brief Description**: This CLI-centric path uses **sysak** and **kdumpctl** for real-time diagnostics. Key subcommands include `sysak memleak` (for slab analysis), `sysak nosched` (for scheduling delays), `sysak iofsstat` (I/O stats), `sysak loadtask` (CPU load), and `sysak mservice` (background monitoring). It also supports crash analysis via `kdumpctl readlog` on Alibaba Cloud Linux 3.

**Key technical facts**:
- Prerequisites: Install SysAK via `sudo yum install -y sysak`; system memory >2 GB for kdump; Alibaba Cloud Linux 3 kernel ≥5.10.134-14 for `readlog`

**When to Use**:
- User needs real-time command-line monitoring of system resources
- Require continuous background monitoring with sysak mservice
- Need to diagnose system crashes after unexpected reboots using kdumpctl
- Looking for specific diagnostic subcommands like memleak for slab analysis or nosched for scheduling delays
- Want to access metrics programmatically via HTTP endpoint at http://127.0.0.1:9200/metrics/raw/

**When NOT to Use**:
- User prefers GUI-based diagnosis over terminal commands
- Need historical OOM event diagnosis for destroyed instances
- Working in environments without sudo access to install SysAK
- Require automated one-click comprehensive diagnosis reports
- Lack required system memory (>2GB) for kdump functionality

**Known Limitations**:
- SysAK is not installed by default and requires manual installation via yum
- kdumpctl readlog feature is specific to Alibaba Cloud Linux 3 (kernel ≥5.10.134-14)
- Requires system memory >2 GB for kdump functionality
- No authentication or cloud credentials support - runs entirely locally
- Limited to supported OS versions: Alibaba Cloud Linux 2/3, Anolis OS 8.4+, or CentOS 7 with kernel ≥3.10 on x86_64

## FAQ

Q: Which path should I start with?
A: If you’re in Chinese mainland or Hong Kong and have `AliyunSysomFullAccess` + `AliyunECSReadOnlyAccess`, start with **SysOM** for safety. Otherwise, use **SysAK** for immediate, scriptable insights without deep kernel knowledge.

Q: What if I need to analyze a past OOM event on a destroyed ECS instance but used the built-in tools path?
A: You’ll hit a dead end—`bpftrace`/`perf` only work on live systems. Only **SysOM** supports **Historical Diagnosis** for destroyed instances (within 14 days).

Q: What if I’m in US West (Silicon Valley) region but chose SysOM for diagnosis?
A: You’ll be unable to access the **Operating System Console**—**SysOM** is only available in **Chinese mainland** and **China (Hong Kong)** regions.

Q: Can I use `sysak memleak` to replace full `crash` + `slabtop` analysis for memory leaks?
A: Partially—it detects slab leaks but lacks the full post-mortem context of `crash` with `kernel-debuginfo`. For deep `SUnreclaim` analysis, the **** path is still superior.

Q: What happens if I lack sudo access but try to use the built-in tools path?
A: You’ll fail to install `bpftrace`, `perf`, or `crash`, and most diagnostic commands (e.g., `ftrace` tracing) require root—making the path unusable without elevated privileges.

Q: Does SysAK provide historical data like SysOM?
A: No—`sysak` is real-time only. It cannot retrieve **Historical Diagnosis** of past OOMs or crashes unless `kdumpctl` captured a dump (and even then, only for recent reboots).

Q: Is there a way to get both GUI ease and deep diagnostics?
A: Not in one tool. Use **SysOM** for initial triage, then switch to **** or **SysAK** if deeper analysis (e.g., `watermark_scale_factor` tuning or `io_uring` queue inspection) is needed.

## Related queries

diagnose system performance, troubleshoot high CPU, memory usage analysis, ksoftirqd latency, container memory discrepancy, fix OOM, analyze I/O delay, scheduling delays, memory fragmentation, TCP congestion, how to check CPU spike, why is memory high, diagnose performance bottleneck, sysak vs bpftr

---
Part of [Alibaba Cloud Linux](https://company-skill.com/p/alinux.md) · https://company-skill.com/llms.txt