含断点与大异常值的时间序列去趋势方法咨询

Hey there, let’s work through this time series challenge you’re dealing with—breakpoints, big outliers, stubborn unit roots, and needing to split out trend and cycle components. I’ve been in this spot before, so let’s break it down step by step.

First things first: those breakpoints and large outliers are almost certainly skewing your ADF test results, and they’ll wreck any standard decomposition you try if you don’t address them first. Let’s start with cleaning up the series.

1. Fix the Outliers & Breakpoints First #

• Detect them properly:
  • In Python, use statsmodels.tsa.stattools.tsoutliers to flag extreme outliers. For structural breaks, the ruptures library or statsmodels.tsa.regime_switching.markov_regression can help identify shift points.
  • In R, the tsoutliers package and strucchange library are your go-tos for outlier and breakpoint detection.
• Adjust the series:
  • For massive outliers, winsorizing (capping values at the 1st/99th percentiles) or replacing with interpolated values (seasonal interpolation works great if you know there’s periodicity) is safer than just deleting them.
  • For breakpoints, either split the series into separate segments at the break points, or add dummy variables for each break in your models to account for level shifts—this keeps the full series intact while controlling for the structural change.

2. Re-Run That ADF Test #

ADF tests are super sensitive to outliers and structural breaks, so it’s no wonder you can’t reject the unit root null right now. Once you’ve cleaned up the series, re-run the ADF test across all those scenarios (with constant, no constant/trend, etc.). If you still can’t reject the unit root, that means your series is genuinely non-stationary, even after accounting for breaks and outliers.

3. Decompose Trend & Cycle (Even for Non-Stationary Series) #

Standard additive/multiplicative decomposition falls apart with non-stationary data, but you’ve got solid options:

• State-Space Models: Use unobserved components models (like statsmodels.tsa.statespace.unobserved_components.UnobservedComponents in Python or StructTS in R). These models explicitly model trend, cycle, noise, and can even incorporate breakpoints/outliers directly into the state structure—no stationarity required upfront.
• STL + Differencing: If you want to stick with a more familiar method, take first differences (since ADF suggests a unit root) to make the series stationary, then apply STL decomposition to the differenced data. You can integrate the trend component back later if you need it on the original scale.
• Filter-Based Methods: The Hodrick-Prescott (HP) filter is common, but be careful with the lambda parameter—too high and you over-smooth. The Baxter-King filter is better for extracting cycles without over-smoothing, but it does require the series to be stationary (so differencing first if needed).

Quick Pro Tips #

• Visualize everything: Plot your raw series, cleaned series, and any decomposition results. A line plot will highlight issues that statistical tests might miss—like a subtle break you didn’t catch.
• Segment if needed: If a breakpoint is really significant, don’t force a single decomposition across the entire series. Fit separate models to each segment—you’ll get way more accurate trend and cycle estimates that way.

内容的提问来源于stack exchange，提问作者PAS