Autograd Functions
4 functions for gradient tracking and computation
Note: These functions enable manual gradient tracking. For automatic backpropagation, see Backpropagation Helpers.
Quick Reference
| Function | Description |
|---|---|
| tensor_requires_grad() | Enable/disable gradient tracking |
| tensor_grad() | Access computed gradients |
| tensor_set_grad() | Manually set gradients |
| tensor_zero_grad() | Reset gradients to zero |
tensor_requires_grad()
tensor_requires_grad(tensor: Tensor, requires_grad: bool) → Tensor
Enables or disables gradient tracking for a tensor. When enabled, operations on this tensor will be tracked for automatic differentiation.
Parameters
| Parameter | Type | Description |
|---|---|---|
| tensor | Tensor | The tensor to modify |
| requires_grad | bool | true to enable tracking, false to disable |
Returns
The modified tensor with gradient tracking enabled/disabled
Example
// Enable gradient tracking for parameters
let weights = tensor_randn([10, 5])
weights = tensor_requires_grad(weights, true)
// Disable gradient tracking for input data
let input_data = tensor([1.0, 2.0, 3.0])
input_data = tensor_requires_grad(input_data, false)
// Now operations on weights will be tracked
let output = tensor_matmul(input_data, weights)Best practice: Enable gradient tracking only for trainable parameters (weights, biases), not for input data.
tensor_grad()
tensor_grad(tensor: Tensor) → [float]
Retrieves the computed gradients for a tensor. Gradients must have been previously computed or set.
Parameters
| Parameter | Type | Description |
|---|---|---|
| tensor | Tensor | Tensor with gradients |
Returns
Array of gradient values matching tensor shape
Example
// After computing gradients via backpropagation
let weights = tensor_randn([5, 3])
weights = tensor_requires_grad(weights, true)
// ... forward pass and backward pass ...
// Retrieve gradients
let grads = tensor_grad(weights)
// Use gradients for optimization
weights = optim_sgd_step(weights, grads, 0.01)Error: Calling this on a tensor without gradients will return zeros or error.
tensor_set_grad()
tensor_set_grad(tensor: Tensor, gradients: [float]) → Tensor
Manually sets gradient values for a tensor. Useful for custom gradient implementations or testing.
Parameters
| Parameter | Type | Description |
|---|---|---|
| tensor | Tensor | Target tensor |
| gradients | [float] | Gradient values (must match tensor size) |
Example
// Manually set gradients for testing
let weights = tensor([1.0, 2.0, 3.0])
weights = tensor_set_grad(weights, [0.1, 0.2, 0.3])
// Verify gradients were set
let grads = tensor_grad(weights)
// grads = [0.1, 0.2, 0.3]Warning: Gradient array must have same total size as tensor. Shape mismatches will cause errors.
tensor_zero_grad()
tensor_zero_grad(tensor: Tensor) → Tensor
Resets all gradients to zero. Essential to call before each backward pass to prevent gradient accumulation.
Parameters
| Parameter | Type | Description |
|---|---|---|
| tensor | Tensor | Tensor to reset |
Returns
Tensor with gradients zeroed
Example
// Training loop pattern
let weights = tensor_randn([10, 5])
weights = tensor_requires_grad(weights, true)
let epochs = 100
let epoch = 0
while epoch < epochs {
// Zero gradients before backward pass
weights = tensor_zero_grad(weights)
// Forward pass
let output = forward(input, weights)
let loss = compute_loss(output, target)
// Backward pass (accumulates gradients)
backward(loss)
// Update weights
let grads = tensor_grad(weights)
weights = optim_sgd_step(weights, grads, 0.01)
epoch = epoch + 1
}Critical: Forgetting to zero gradients causes gradient accumulation, leading to incorrect training.
Complete Example
Full gradient tracking workflow:
// Initialize parameters with gradient tracking
let w = tensor_randn([3, 1])
w = tensor_requires_grad(w, true)
let b = tensor_zeros([1])
b = tensor_requires_grad(b, true)
// Training data
let x = tensor([1.0, 2.0, 3.0])
let y = tensor([7.0]) // True value
let lr = 0.01
let epochs = 100
let epoch = 0
while epoch < epochs {
// Zero gradients
w = tensor_zero_grad(w)
b = tensor_zero_grad(b)
// Forward pass
let pred = nn_linear(x, w, b)
let loss = nn_mse_loss(pred, y)
// Backward pass (manually compute gradients)
let grad_pred = autograd_compute_mse_grad(pred, y)
let linear_grads = autograd_compute_linear_grad(x, w, b, grad_pred)
// Set gradients
w = tensor_set_grad(w, linear_grads.1)
b = tensor_set_grad(b, linear_grads.2)
// Retrieve and apply gradients
let w_grad = tensor_grad(w)
let b_grad = tensor_grad(b)
w = optim_sgd_step(w, w_grad, lr)
b = optim_sgd_step(b, b_grad, lr)
epoch = epoch + 1
}
print("Training complete!")