The Three Attributes That Define a Tensor

Every tensor is fully described by three things. If you can rattle these off and explain how each one bites you in practice, you've cleared the phone-screen bar.

The Classic: view vs reshape

This is the most-asked PyTorch tensor question. The answer hinges on contiguity.

x = torch.arange(6).reshape(2, 3)
y = x.T                 # transpose: a VIEW

y.view(6)             # RuntimeError!
y.reshape(6)          # OK — silently copies
y.contiguous().view(6) # OK — explicit copy

Rapid-Fire Conceptual Round

Broadcasting — Explain It in One Sentence

Interviewers love: "walk me through the broadcasting rules." The one-liner they want: align shapes from the right; each dimension must be equal or one of them must be 1; size-1 dims are virtually expanded.

a = torch.arange(3).reshape(3, 1)  # (3,1)
b = torch.arange(4).reshape(1, 4)  # (1,4)
c = a + b                          # (3,4)

Spot the Bug: Device Mismatch

You're asked to review a colleague's inference snippet on a GPU box. It crashes. Click the line that causes the runtime error.

device = (torch.accelerator.current_accelerator()
          if torch.accelerator.is_available() else "cpu")
x = x.to(device)   # .to() returns a NEW tensor — must reassign

In-Place Ops: The Trap

Operations ending in _ (add_, mul_, relu_) mutate in place. They save memory — and can silently corrupt the autograd graph (Week 2). Interviewers use this to bridge into gradients.

The Graph Is Built on Every Forward Pass

Unlike TensorFlow 1.x's static graph, PyTorch is define-by-run. This is why a plain Python if works inside forward() — the graph simply records whichever branch ran.

w = torch.randn(5, 3, requires_grad=True)
b = torch.randn(3, requires_grad=True)
x = torch.ones(5)

z = x @ w + b           # graph recorded here
loss = z.sum()
loss.backward()         # reverse pass

print(w.grad)           # populated

Watch the Graph Think

Here's how the forward tensors, the loss, and the backward engine coordinate. Step through it — this is exactly the narration you'd give on a whiteboard.

Three Ways to Stop Tracking — Know the Differences

"What's the difference between no_grad, inference_mode, and detach?" is a staple. Here's the crisp answer:

Spot the Bug: The Silent Killer

This loop "trains" but the model barely learns. Click the missing/wrong line.

Conceptual Round

Define a Model — the Registration Rules

class Net(nn.Module):
  def __init__(self):
    super().__init__()   # MUST call first
    self.net = nn.Sequential(
      nn.Linear(784, 256), nn.ReLU(),
      nn.Linear(256, 10))  # raw logits

  def forward(self, x):
    return self.net(x)   # no softmax!

The Canonical Loop — Live-Coding Answer

Memorize this. The five steps, in order:

model.train()                     # dropout/BN in train mode
for x, y in loader:
  x, y = x.to(device), y.to(device)
  pred = model(x)                 # 1
  loss = loss_fn(pred, y)         # 2
  loss.backward()                 # 3
  optimizer.step()                # 4
  optimizer.zero_grad()           # 5

Spot the Bug: The Loop That Won't Learn

A candidate wrote this in a screen. It compiles and runs — but the loss never drops. Click the offending line.

Losses & Optimizers — the Gotchas

Conceptual Round

Dataset vs DataLoader — the Division of Labor

The Knobs That Come Up

loader = DataLoader(
  ds, batch_size=64, shuffle=True,
  num_workers=4,
  pin_memory=True,
  persistent_workers=True,
  drop_last=True)

x = x.to(device, non_blocking=True)

Transforms: Know the Deprecation

A subtle "are you current?" signal: the modern API is torchvision.transforms.v2, and ToTensor is deprecated.

from torchvision.transforms import v2

tf = v2.Compose([
  v2.ToImage(),
  v2.RandomResizedCrop(224, antialias=True),
  v2.ToDtype(torch.float32, scale=True),
  v2.Normalize(mean=..., std=...)])

Spot the Bug: Duplicated Augmentations

Training with num_workers=4 and random augmentation. Every 4th sample looks augmented identically. Click the root cause.

def seed_worker(worker_id):
  s = torch.initial_seed() % 2**32
  np.random.seed(s); random.seed(s)
g = torch.Generator(); g.manual_seed(0)
DataLoader(ds, num_workers=4, worker_init_fn=seed_worker, generator=g)

Checkpointing — Conceptual Round

BatchNorm vs LayerNorm — the One-Liner

The answer they want: "Use LayerNorm for sequence models/Transformers or when batches are small/variable, because it doesn't depend on batch statistics."

Match the Layer to Its Job

Drag each layer onto the use-case an interviewer would expect it for.

Transfer Learning — the Modern API

Interviewers spot outdated candidates instantly: pretrained=True is deprecated. Use the Weights enum.

from torchvision.models import resnet18, ResNet18_Weights

w = ResNet18_Weights.DEFAULT
model = resnet18(weights=w)
preprocess = w.transforms()   # exact preprocessing

for p in model.parameters():
  p.requires_grad = False   # freeze FIRST
model.fc = nn.Linear(model.fc.in_features, n)  # then replace
opt = torch.optim.SGD(model.fc.parameters(), lr=1e-3)

Spot the Bug: Nothing Trains

A candidate's fine-tuning code runs, loss is flat, accuracy never moves. Click the line responsible.

Conceptual Round

Why GradScaler Exists

AMP runs eligible ops in fp16/bf16. But fp16 has a narrow dynamic range — small gradients underflow to zero. The GradScaler multiplies the loss by a large factor so gradients stay representable, then unscales before the optimizer step.

The Correct AMP Loop (Current API)

from torch.amp import autocast, GradScaler
scaler = GradScaler("cuda")

for x, y in loader:
  optimizer.zero_grad()
  with autocast(device_type="cuda"):
    out = model(x)          # forward + loss ONLY
    loss = loss_fn(out, y)
  scaler.scale(loss).backward()   # OUTSIDE autocast
  scaler.step(optimizer)
  scaler.update()

Spot the Bug: Broken AMP

This AMP loop produces NaNs and gradient clipping seems to do nothing. Click the line at fault.

scaler.scale(loss).backward()
scaler.unscale_(optimizer)              # unscale FIRST
clip_grad_norm_(model.parameters(), 1.0)
scaler.step(optimizer); scaler.update()

The fp16-vs-bf16 Question

Profiling — Show You Can Measure

"How would you find the bottleneck?" Reach for torch.profiler with a schedule that skips warmup.

from torch.profiler import profile, schedule, ProfilerActivity

with profile(
  activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
  schedule=schedule(wait=1, warmup=1, active=3)) as p:
  for _ in range(5):
    model(inputs); p.step()
print(p.key_averages().table(sort_by="cuda_time_total"))

The Compile Pipeline in One Breath

model = torch.compile(model)              # one line
model = torch.compile(model, mode="max-autotune")
model = torch.compile(model, fullgraph=True)  # no graph breaks

Spot the Bug: Slower After Compiling

A candidate wraps their model in torch.compile and it's slower than eager, recompiling constantly. Click the cause.

The Decision: DDP vs FSDP

from torch.distributed.fsdp import fully_shard
for layer in model.layers:
  fully_shard(layer)      # shard submodules first
fully_shard(model)        # then the root
opt = torch.optim.Adam(model.parameters())  # AFTER sharding!

Watch DDP Ranks Sync

Step through one training step across two GPUs — this is the narration you'd give when asked "how does DDP keep replicas in sync?"

Conceptual Round

Custom Ops: The Compiler Contract

The modern way to insert a C++/CUDA/Triton kernel is torch.library.custom_op. The key insight interviewers want: torch.compile treats a custom op as an opaque black box, so you must hand it the metadata it can't infer — chiefly a fake kernel.

@torch.library.custom_op("mylib::my_op",
    mutates_args=())
def my_op(x: Tensor) -> Tensor:
  return my_cuda_kernel(x)

@my_op.register_fake
def _(x):
  return torch.empty_like(x)  # metadata MUST match

Export, Deployment & Memory Levers

System Design #1: Train a 70B Model

"Design distributed training for a 70B-parameter LLM on a cluster of 8-GPU nodes." Here's the rubric a strong candidate hits:

System Design #2: Serve an LLM at Low Latency

"Serve this model for real-time inference with tight p99 latency." The rubric:

Coding: A Custom autograd Function

"Implement a custom backward." The modern pattern splits pure compute (forward) from context setup (setup_context):

class MyReLU(Function):
  @staticmethod
  def forward(x):
    return x.clamp(min=0)
  @staticmethod
  def setup_context(ctx, inputs, output):
    ctx.save_for_backward(inputs[0])
  @staticmethod
  def backward(ctx, grad_out):
    x, = ctx.saved_tensors
    return grad_out * (x > 0)