Optimization with Optim.jl

FastInterpolations.jl integrates seamlessly with Optim.jl for optimization problems over interpolated surfaces.

This guide demonstrates three approaches to provide derivatives, ranging from simplest to fastest.

Setup: Rosenbrock on an Interpolated Surface

The classic Rosenbrock function $f(x,y) = (1-x)^2 + 100(y-x^2)^2$ is a standard optimization benchmark. Here we build a cubic interpolant over a 2D grid and optimize it:

using FastInterpolations, Optim

rosenbrock(x, y) = (1.0 - x)^2 + 100.0 * (y - x^2)^2

# Build the interpolated surface
xg = range(-0.7, 1.5, length=101)
yg = range(-0.7, 1.5, length=101)
zg = [rosenbrock(xi, yi) for xi in xg, yi in yg]

itp = cubic_interp((xg, yg), zg; extrap=:extension, bc=CubicFit())

Why `extrap=:extension`?

Trust-region and line-search methods may evaluate points outside the grid during intermediate steps. :extension extrapolates smoothly from the boundary, preventing errors without distorting the objective landscape near the boundary.

Three Ways to Run Optimization

using Optim: ADTypes

x0 = [-0.15, 0.6] # Initial guess

# ── Method 1: Default (Optim estimates derivatives via finite differences) ──
result = optimize(itp, x0, NewtonTrustRegion())

# ── Method 2: AD packages (machine-precision gradients) ──
# FastInterpolations.jl supports all major AD packages
using ForwardDiff, Zygote, Enzyme 
result = optimize(itp, x0, NewtonTrustRegion(), autodiff=ADTypes.AutoForwardDiff())
result = optimize(itp, x0, NewtonTrustRegion(), autodiff=ADTypes.AutoZygote())
result = optimize(itp, x0, NewtonTrustRegion(), autodiff=ADTypes.AutoEnzyme())

# ── Method 3: Analytical derivatives (fastest & accurate) ──
grad!(G, x) = FastInterpolations.gradient!(G, itp, x)
hess!(H, x) = FastInterpolations.hessian!(H, itp, x)
result = optimize(itp, grad!, hess!, x0, NewtonTrustRegion())

Method	Derivatives	Accuracy
Default	Numerical (Finite Difference)	~O(h²)
AD (ForwardDiff, Zygote, Enzyme)	Automatic Differentiation	Accurate
Analytical `gradient!`/`hessian!`	Direct Analytic Estimation	Accurate

Method 1 requires zero setup — Optim.jl internally approximates derivatives via finite differences when no autodiff or gradient function is provided, but numerical approximation can be inaccurate or unstable depending on the surface. Methods 2 and 3 both provide exact derivatives of the interpolant (identical to machine precision); Method 3 is recommended for performance-critical loops.

In-place convention

gradient! and hessian! follow Optim.jl's optimize(f, g!, h!, x0, method) convention — they mutate the first argument in-place, avoiding allocations in the optimization loop.

Results

FDM vs Analytical Newton on Rosenbrock

The plot compares Method 1 (finite differences), Method 2 (AD; ForwardDiff) and Method 3 (analytical derivatives) on the Rosenbrock surface. Method 2 & 3 converge faster and more stably. Note that all methods converge to the minimum of the interpolated surface, which may differ slightly from the true analytic minimum at (1, 1) — this is inherent to optimizing over a discrete grid representation.

The full runnable script is available at examples/rosenbrock_optim.jl.