# Diagnosing Hangings and Deadlocks in Multi-Node Multi-GPU Python Programs

**Moved to [Debugging PyTorch Programs](./pytorch.md#diagnosing-crashes-hangs-and-tracing-execution).**

## Helper tools

**Moved to [Debugging PyTorch Programs](./pytorch.md#helper-tools).**

## Approaches to diagnosing multi-gpu hanging / deadlocks

**Moved to [Debugging PyTorch Programs](./pytorch.md#approaches-to-diagnosing-multi-gpu-hanging--deadlocks).**

### py-spy

**Moved to [Debugging PyTorch Programs](./pytorch.md#py-spy).**

#### multi-process py-spy

**Moved to [Debugging PyTorch Programs](./pytorch.md#multi-process-py-spy).**

#### multi-node py-spy via srun

**Moved to [Debugging PyTorch Programs](./pytorch.md#multi-node-py-spy-via-srun).**

#### multi-node py-spy via pdsh

**Moved to [Debugging PyTorch Programs](./pytorch.md#multi-node-py-spy-via-pdsh).**

#### multi-node py-spy via ds_ssh

**Moved to [Debugging PyTorch Programs](./pytorch.md#multi-node-py-spy-via-ds_ssh).**

### Network-level hanging

**Moved to [Debugging PyTorch Programs](./pytorch.md#network-level-hanging).**

### Isolate problematic GPUs

**Moved to [Debugging PyTorch Programs](./pytorch.md#isolate-problematic-gpus).**

### python `trace`

**Moved to [Debugging PyTorch Programs](./pytorch.md#python-trace).**

#### NicerTrace

**Moved to [Debugging PyTorch Programs](./pytorch.md#nicertrace).**

#### Working with generated trace files

**Moved to [Debugging PyTorch Programs](./pytorch.md#working-with-generated-trace-files).**

### good old `print`

**Moved to [Debugging PyTorch Programs](./pytorch.md#good-old-print).**

### core files

**Moved to [Debugging PyTorch Programs](./pytorch.md#core-files).**

### Code loops

**Moved to [Debugging PyTorch Programs](./pytorch.md#code-loops).**

## Hardware-specific issues

**Moved to [Debugging PyTorch Programs](./pytorch.md#hardware-specific-issues).**

### AMD/ROCm hangs or slow with IOMMU

**See [Troubleshooting AMD GPUs](../compute/accelerator/amd/debug.md#hangs-or-slow-multi-gpu-runs-and-iommu).**