Question

What are the differences between autoregressive and moving average models? Consider the following: model specifications, stationarity, the shapes of their autocorrelation and partial autocorrelation functions.

Answer 1

An autoregressive model is any model that tries to predict the next value of a series based on past values alone. A moving average is one sort of autoregressive model. It will work well if the underlying process is random variation around a mean, and the mean drifts only slowly compared to the sample frequency.

Partial autocorrelation function

The partial autocorrelation is a tool for the identification and estimation of the ARIMA model. It is defined as the amount of correlation between two variables that is not explained by their mutual correlations with a given set of other variables.

Partial autocorrelation at lag kk is defined as the autocorrelation between xtxt and xt−kxt−k that is not accounted for by lags 1 through to kk-1, which means that correlations with all the elements up to lag kk are removed. Following this definition, a partial autocorrelation for lag 1 is equivalent to an autocorrelation.

The partial autocorrelation function (PACF) is the set of partial autocorrelation coefficients (k)(k) arranged as a function of kk. This function can be used to detect the presence of an autoregressive process in time series and identify the order of this process. Theoretically, the number of significant lags determines the order of the autoregressive process.

Auto-correlation of continuous-time signalEdit

Given a signal {\displaystyle f(t)}, the continuous autocorrelation {\displaystyle R_{ff}(\tau )} is most often defined as the continuous cross-correlation integral of {\displaystyle f(t)} with itself, at lag {\displaystyle \tau }.[1]:p.411

{\displaystyle R_{ff}(\tau )=\int _{-\infty }^{\infty }f(t+\tau ){\overline {f(t)}}\,{\rm {d}}t=\int _{-\infty }^{\infty }f(t){\overline {f(t-\tau )}}\,{\rm {d}}t}

(Eq.6)

where {\displaystyle {\overline {f(t)}}} represents the complex conjugate of {\displaystyle f(t)}. Note that the parameter {\displaystyle t} in the integral is a dummy variable and is only necessary to calculate the integral. It has no specific meaning.

Auto-correlation of discrete-time signalEdit

The discrete autocorrelation {\displaystyle R} at lag {\displaystyle \ell } for a discrete-time signal {\displaystyle y(n)} is

{\displaystyle R_{yy}(\ell )=\sum _{n\in Z}y(n)\,{\overline {y(n-\ell )}}}

(Eq.7)

The above definitions work for signals that are square integrable, or square summable, that is, of finite energy. Signals that "last forever" are treated instead as random processes, in which case different definitions are needed, based on expected values. For wide-sense-stationary random processes, the autocorrelations are defined as

{\displaystyle R_{ff}(\tau )=\operatorname {E} \left[f(t){\overline {f(t-\tau )}}\right]}

{\displaystyle R_{yy}(\ell )=\operatorname {E} \left[y(n)\,{\overline {y(n-\ell )}}\right].}

For processes that are not stationary, these will also be functions of {\displaystyle t}, or {\displaystyle n}.

For processes that are also ergodic, the expectation can be replaced by the limit of a time average. The autocorrelation of an ergodic process is sometimes defined as or equated to[4]

{\displaystyle R_{ff}(\tau )=\lim _{T\rightarrow \infty }{\frac {1}{T}}\int _{0}^{T}f(t+\tau ){\overline {f(t)}}\,{\rm {d}}t}

{\displaystyle R_{yy}(\ell )=\lim _{N\rightarrow \infty }{\frac {1}{N}}\sum _{n=0}^{N-1}y(n)\,{\overline {y(n-\ell )}}.}

These definitions have the advantage that they give sensible well-defined single-parameter results for periodic functions, even when those functions are not the output of stationary ergodic processes.

Alternatively, signals that last forever can be treated by a short-time autocorrelation function analysis, using finite time integrals. (See short-time Fourier transform for a related process.)

Definition for periodic signalsEdit

If {\displaystyle f} is a continuous periodic functions of period {\displaystyle T}, the integration from {\displaystyle -\infty } to {\displaystyle \infty } is replaced by integration over any interval {\displaystyle [t_{0},t_{0}+T]} of length {\displaystyle T}:

{\displaystyle R_{ff}(\tau )\triangleq \int _{t_{0}}^{t_{0}+T}f(t+\tau ){\overline {f(t)}}\,dt}

which is equivalent to

{\displaystyle R_{ff}(\tau )\triangleq \int _{t_{0}}^{t_{0}+T}f(t){\overline {f(t-\tau )}}\,dt}

PropertiesEdit

In the following, we will describe properties of one-dimensional autocorrelations only, since most properties are easily transferred from the one-dimensional case to the multi-dimensional cases. These properties hold for wide-sense stationary processes.[5]

• A fundamental property of the autocorrelation is symmetry, {\displaystyle R_{ff}(\tau )=R_{ff}(-\tau )}, which is easy to prove from the definition. In the continuous case,

the autocorrelation is an even function

{\displaystyle R_{ff}(-\tau )=R_{ff}(\tau )\,} when {\displaystyle f} is a real function,

and the autocorrelation is a Hermitian function

{\displaystyle R_{ff}(-\tau )=R_{ff}^{*}(\tau )\,} when {\displaystyle f} is a complex function.

• The continuous autocorrelation function reaches its peak at the origin, where it takes a real value, i.e. for any delay {\displaystyle \tau }, {\displaystyle |R_{ff}(\tau )|\leq R_{ff}(0)}.[1]:p.410 This is a consequence of the rearrangement inequality. The same result holds in the discrete case.
• The autocorrelation of a periodic function is, itself, periodic with the same period.
• The autocorrelation of the sum of two completely uncorrelated functions (the cross-correlation is zero for all {\displaystyle \tau }) is the sum of the autocorrelations of each function separately.
• Since autocorrelation is a specific type of cross-correlation, it maintains all the properties of cross-correlation.
• By using the symbol {\displaystyle *} to represent convolution and {\displaystyle g_{-1}} is a function which manipulates the function {\displaystyle f} and is defined as {\displaystyle g_{-1}(f)(t)=f(-t)}, the definition for {\displaystyle R_{ff}(\tau )} may be written as:

{\displaystyle R_{ff}(\tau )=(f*g_{-1}({\overline {f}}))(\tau )}