# IronWolf PBS — PERC H710 Drive Failure Diagnosis

## Server Info
- **Hostname:** IronWolf
- **IP:** 192.168.0.45
- **Service Tag:** 9JBC942
- **RAID Controller:** Dell PERC H710 (LSI MegaRAID SAS)
- **OS:** Proxmox Backup Server (Debian Trixie)

## Situation
Amber drive light on front of server. Need to identify which physical drive is failed before pulling anything.

OS-level check shows both virtual disks as "running" — the PERC is hiding the physical failure from the OS. Need perccli to query physical disk status.

---

## Step 1 — Download perccli on a working machine

Go to **dell.com/support**, enter service tag **9JBC942**, go to **Drivers & Downloads**, filter by **SAS RAID**, download the Linux PERCCLI tar.gz.

Or download directly on a Mac/Windows machine and SCP it over.

The archive contains a `.deb` file inside — extract it before transferring.

---

## Step 2 — Transfer and install perccli on IronWolf

From your laptop on the home network:

```bash
scp perccli_*.deb root@192.168.0.45:/tmp/
```

Then on IronWolf:

```bash
dpkg -i /tmp/perccli_*.deb
```

---

## Step 3 — Check physical disk status

```bash
/opt/MegaRAID/perccli/perccli64 /c0 /eall /sall show
```

This shows every physical slot, drive state (Online/Failed/Unconfigured Bad), and bay number.

Look for any drive showing **Failed** or **Unconfigured Bad** in the State column.

---

## Step 4 — Identify the exact slot

```bash
/opt/MegaRAID/perccli/perccli64 /c0 show all
```

This shows the full controller status including virtual disk health (Optimal/Degraded).

---

## Step 5 — Locate the drive physically

Once you have the slot number from perccli, you can blink the drive LED to confirm:

```bash
/opt/MegaRAID/perccli/perccli64 /c0 /e<enclosure> /s<slot> start locate
```

The drive's LED will blink. To stop blinking:

```bash
/opt/MegaRAID/perccli/perccli64 /c0 /e<enclosure> /s<slot> stop locate
```

Replace `<enclosure>` and `<slot>` with the values from Step 3.

---

## Step 6 — After replacing the drive

Once the new drive is inserted, check rebuild status:

```bash
/opt/MegaRAID/perccli/perccli64 /c0 /eall /sall show rebuild
```

The PERC H710 will automatically start rebuilding. Monitor progress:

```bash
watch -n 30 '/opt/MegaRAID/perccli/perccli64 /c0 /eall /sall show rebuild'
```

---

## What we already know
- Both virtual disks (`/dev/sda` 4TB, `/dev/sdb` 1TB) show state: **running**
- No kernel-level RAID errors in dmesg
- PBS backups are continuing normally
- Physical failure is being masked by the PERC controller
- Suspected failed drive: **front left bay** (unconfirmed — use perccli to verify before pulling)

---

## SSH into IronWolf
```bash
ssh root@192.168.0.45
```