多项式回归拟合sin(x)失效：代价函数提前收敛问题咨询

Hey there! Let's dig into why you're hitting this wall with polynomial regression on sin(x) when it worked so well for sqrt(x). The short answer is: polynomial regression can fit periodic functions like sin(x) in limited ranges, but there are key pitfalls and better approaches to consider. Here's a breakdown:

1. Is polynomial regression inherently unsuitable for periodic data? #

Not exactly—but it's not the most efficient or natural choice. Polynomials are non-periodic, so they can only approximate periodic functions over a finite interval. For example, a high-enough order polynomial can mimic sin(x) between, say, -π and π, but will blow up outside that range.

Compare that to sqrt(x), which is a smooth, monotonic function—low-to-moderate order polynomials can capture its shape easily without needing extreme complexity. For sin(x)'s oscillating pattern, you'll need a much higher degree polynomial to even get close, and even then, you might run into overfitting issues.

2. Why is your cost function stopping early? #

The "触底不再优化" behavior usually points to one of these issues with your setup:

• Too low polynomial degree: If you're using the same low order that worked for sqrt(x) (like 2-3 degrees), the model simply doesn't have enough capacity to capture sin(x)'s oscillations. It hits a local minimum where no further adjustments to theta can reduce the cost—because the model can't do better.
• Poor feature scaling: Polynomial terms (like x^5, x^10) get extremely large as x moves away from 0. This can make gradient descent slow to converge, or get stuck in a flat region of the cost surface. sqrt(x) likely has a smaller input range or less extreme high-order terms, so scaling wasn't an issue there.
• Suboptimal learning rate: If your learning rate is too small, gradient descent might crawl to a halt before finding a better minimum. If it's too large, you might see oscillations, but if you're seeing a hard stop, a too-small rate or model capacity limit is more likely.

3. Is your hypothesis function the problem? #

Your basic polynomial hypothesis (hθ(x) = θ₀ + θ₁x + θ₂x² + ... + θₙxⁿ) isn't "wrong"—but it's not the best tool for this job. Here's how to fix or replace it:

• Boost the polynomial degree: Try increasing the degree to 5-10 (or higher) and see if the cost function continues to decrease. Just be aware that higher degrees mean higher risk of overfitting—add L1/L2 regularization to keep things in check.
• Add trigonometric features: Instead of relying solely on polynomial terms, build a hypothesis that includes sin(x), cos(x), sin(2x), cos(2x), etc. This is essentially a Fourier series approach, and it's way more efficient at capturing periodic patterns. For example:

  hθ(x) = θ₀ + θ₁sin(x) + θ₂cos(x) + θ₃sin(2x) + θ₄cos(2x) + ...
  

  This is still a linear regression model (since the parameters θ are linear), and it will fit sin(x) perfectly with just the θ₁sin(x) term!
• Normalize your inputs: Scale your x values to a small range (like [-1, 1] or standardized to mean 0, variance 1) to prevent high-order polynomial terms from dominating the gradient calculations.

4. Quick next steps to test #

• First, check your polynomial degree—crank it up and see if the cost function starts improving.
• Add feature scaling if you haven't already; this is critical for high-degree polynomials.
• Experiment with trigonometric features instead of pure polynomials—you'll likely see a huge jump in performance.
• If you're using vanilla gradient descent, try switching to an optimized algorithm like Adam or L-BFGS, which can handle tricky cost surfaces better.

内容的提问来源于stack exchange，提问作者MikeG