Numerical Stability: Log-Sum-Exp

easy (<10 mins) optimization deep-learning numerical-stability logarithms probability

this year by E

When dealing with probabilities, especially in log-space, sums of exponentials can lead to numerical underflow or overflow. For example, computing $\log (\sum_{i} \exp (x_{i}))$ can be problematic if $x_{i}$ are very large (overflow) or very small (underflow). The "Log-Sum-Exp" trick provides a numerically stable way to compute this value.

The trick is based on the identity:

\log (\sum_{i} \exp (x_{i})) = α + \log (\sum_{i} \exp (x_{i} - α))

where $α = max (x_{1}, \dots, x_{N})$ . By subtracting the maximum value, the exponents $x_{i} - α$ become non-positive, preventing overflow. Furthermore, at least one term $\exp (x_{k} - α)$ will be $\exp (0) = 1$ , preventing all terms from vanishing to zero due to underflow.

Your task is to implement the logsumexp function using this trick.

Implementation Details: * logsumexp(x): Takes a 1D NumPy array x of log-probabilities or log-unnormalized scores. * It should return a single scalar representing the numerically stable $\log (\sum_{i} \exp (x_{i}))$ .

Verification: 1. Test with a simple array x = np.array([1, 2, 3]). 2. Test with an array containing large values, e.g., x = np.array([1000, 1001, 1002]). 3. Test with an array containing small (negative) values, e.g., x = np.array([-1000, -1001, -1002]). 4. Compare your results with scipy.special.logsumexp for various inputs. Ensure the output is very close (e.g., difference less than 1e-10).