# Changelog

All changes we make to the assignment code or PDF will be documented in this file.

## [26.1.4] - 2026-05-01
- code: sharded model tests use tensors with even leading dimensions, allowing alternate solutions

## [26.1.3] - 2026-04-28
- writeup: write proper FA3-style backward pass with two passes, one for dK, dV, and a separate for dQ
- writeup: enforce consistency in backward and forward algorithm notation
- writeup: encourage using the DDP test for the DDP implementation

## [26.1.2] - 2026-04-28
- code: fix package existence bug for `__init__.py`
- code: relax equality for FSDP testing
- writeup: correct test file names for DDP

## [26.1.1] - 2026-04-20
- handout: Remove H100-specific references in favor of B200 use and TMA description

## [26.1.0] - 2026-04-16
- handout: clean up language, fix a few inconsistencies
- handout: fix triton code compile issue
- handout: reduce work for profiling bits

## [26.0.0] - 2026-04-15
- handout: more detailed Nsight Systems profiler
- handout: replace leaderboard task with full forward and backward pass on two B200s
- handout: add section on activation checkpointing
- handout: remove excessive benchmarking, multi-GPU sweeps
- handout: add FSDP section
- handout: remove bucketed DDP


## [1.0.5] - 2025-07-03
### Fixed
- code: typos regarding FlashAttention2 in the adapters

### Added
- code: uv build backend for building the package


## [1.0.4] - 2025-04-27

### Fixed
- handout: details for leaderboard submission
- handout: replace Pytorch profiler with Nsight for DDP

### Added
- handout: clarify which attention implementations to benchmark

## [1.0.3] - 2025-04-24
### Fixed
- handout: `memory_profiling` (b) should sweep over context length, not model size
- handout: keyword argument in FA2 Triton starter code from `tile_shape` -> `block_shape`
- handout: minor typo in `flash_backward` problem statement
- handout: launch all-gather example using `uv run`

## [1.0.2] - 2025-04-22
### Added
- handout: clarify interface for flash autograd function
- handout: clarify submission for attention benchmarking

### Fixed
- handout: fix small notation issues in the flash algorithm
- handout: fix math error and explanation for flash backward savings
- handout: change parameters used for attention benchmarking to be sensible

### Removed
- handout: some memory benchmarking that is incompatible with modern PyTorch

## [1.0.1] - 2025-04-17
### Added
- code: Include tests for logsumexp in flash forward implementation
- handout: Clarify interface for flash autograd function
- code: Test causal=True for forward as well as backward

## [1.0.0] - 2025-04-16

### Added
- handout/code: add FlashAttention2
- handout: add proper profiling with Nsight Systems
- handout: add communication accounting
- code: tests for additional content

### Changed
- handout: greatly improve demo example for Triton
- handout: remove busywork for communication

### Fixed

- handout: clarify that `ddp_bucketed_benchmarking` doesn't require the full
  grid of runs.

## [0.0.4] - 2024-04-23

### Added

### Changed

- code: remove try-finally blocks in DDP tests.

### Fixed

- handout: remove outdated mention of a problem that doesn't exist on the assignment
- handout: fix Slurm environment variables in examples.
- handout: clarify assumptions in `ddp_bucketed_benchmarking` (b).

## [0.0.3] - 2024-04-21

### Added

### Changed

- code: remove `humanfriendly` from requirements.txt, add `matplotlib`
- handout: modify problem `distributed_communication_multi_node` to specify that
  multinode measurements should be 2x1, 2x2, and 2x3.
- handout: clarify that `torch.cuda.synchronize()` is necessary for timing
  collective communication ops, even when they are called with `async_op=False`.

### Fixed

- handout: fixed cut off text in problem memory_profiling (a)
- handout: fixed mismatch between slurm config and description text in section 3.2
- code: fix `ToyModelWithTiedWeights` to actually tie weights.
- handout: fix typo in bucketed DDP test command, should be `pytest tests/test_ddp.py` 
- handout: fix deliverable of `ddp_overlap_individual_parameters_benchmarking`
  (a) to not ask for communication time, only end-to-end step time.
- handout: clarify analysis in `optimizer_state_sharding_accounting` (a).

## [0.0.1] - 2024-04-17

### Added

- handout: added a short question about variability on problem benchmarking_script

### Changed

### Fixed

- handout: fixed typo in problem `triton_rmsnorm_forward`. The adapters should
  return the classes, not the `.apply` attribute.
- code: added `-e` flag to `./cs336-systems/'[test]'`
- handout: clarified recommendation about the timeit module
- handout: clarified question about kernel with highest CUDA total

## [0.0.0] - 2024-04-16

Initial release.