Error Statistics for the Survey of Professional Forecasters for Nonfarm Payroll Employment
Release Date: 08/29/2021
Tom Stark
Assistant Vice President | Assistant Director
Real-Time Data Research Center
Economic Research Department
Federal Reserve Bank of Philadelphia
Source for Historical Realizations: Bureau of Labor Statistics via Haver Analytics
1. OVERVIEW.
This document reports error statistics for median projections from the Survey of Professional
Forecasters (SPF), conducted since 1990 by the Federal Reserve Bank of Philadelphia. We provide
the results in a series of tables and, in the PDF version of this document, a number of charts. The
tables show the survey variable forecast and, importantly, the transformation of the data that we used to
generate the statistics. (The transformation is usually a quarter-over-quarter growth rate, expressed
in annualized percentage points. However, some variables, such as interest rates, the unemployment rate,
and housing starts are untransformed and, thus, expressed in their natural units.)
The paragraphs below explain the format of the tables and charts and the methods used to compute the
statistics. These paragraphs are general. The same discussion applies to all variables in the survey.
2. DESCRIPTION OF TABLES.
Tables 1A-1B report error statistics for various forecast horizons, sample periods, and choices of the
real-time historical value that we used to assess accuracy. In each quarterly survey, we ask our
panelists for their projections for the current quarter and the next four quarters. The current quarter
is defined as the quarter in which we conducted the survey. Our tables provide error statistics separately
for each quarter of this five-quarter horizon, beginning with the current quarter (denoted H = 1) and ending
with the quarter that is four quarters in the future (H = 5). For each horizon, we report the mean forecast
error [ME(S)], the mean absolute forecast error [MAE(S)], and the root-mean-square error [RMSE(S)].
All are standard measures of accuracy, though the academic literature generally places the most weight on
the latter.
We define a forecast error as the difference between the historical value and the forecast. The mean error
for each horizon is simply the average of the forecast errors at that horizon, constructed over the sample
periods shown in Table 1A. Other things the same, a forecast with a mean error close to zero is better than
one with a mean error far from zero. The mean absolute error is the sample average of the absolute value
of the errors. Many analysts prefer this measure to the mean error because it does not allow large positive
errors to offset large negative errors. In this sense, the mean absolute error gives a cleaner estimate
of the size of the errors. Decision makers, however, may care not only about the average size of the
errors but also about their variability, as measured by variance. Our last measure of accuracy is one that
reflects the influence of the mean error and the variance of the error. The root-mean-square error for
the SPF [RMSE(S)], the measure most often used by analysts and academicians, is the square root of the
the average squared error. The lower the root-mean-square error, the more accurate the forecast.
2.1. Benchmark Models.
The forecast error statistics from the SPF are of interest in their own right. However, it is often more
interesting to compare such statistics with those of alternative, or benchmark, forecasts. Tables 1A-1B
report four such comparisons. They show the ratio of the root-mean-square error of the SPF forecast to that
of four benchmark models. The benchmark models are statistical equations that we estimate on the data.
We use the equations to generate projections for the same horizons included in the survey. In effect, we
imagine standing back in time at each date when a survey was conducted and generating a separate forecast
with each benchmark model. We do this in the same way that a survey panelist would have done using his own
model.
Table 1A reports the root-mean-square-error ratios using as many observations as possible for each model.
The number of observations can differ from model to model. We first compute the RMSE for each model. We
then construct the ratio.
Table 1B reports RMSE ratios after we adjust the samples to include only the observations common to
both models in the pair. Accordingly, the ratios reported in Table 1B may differ slightly from
those of Table 1A, depending on the availability of sufficient real-time observations for estimating
the benchmark models or for computing the errors of the SPF or benchmark forecasts. Table 1B also reports
three two-sided p-values for each ratio. The p-values, corrected for the presence of heteroskedasticity
and serial correlation in the time series of differences in squared forecast errors, are those for
the test of equality of mean-square error between the SPF and the benchmark. The p-values are those for:
(1) The Diebold-Mariano statistic (July 1995, Journal of Business and Economic Statistics), using a
uniform lag window with the truncation lag set to the forecast horizon minus unity. When the
uniform lag window produces a nonpositive standard error, the Bartlett window is used.
(2) The Harvey-Leybourne-Newbold correction (1997, International Journal of Forecasting) to the
Diebold-Mariano statistic.
(3) The Diebold-Mariano statistic, using a Bartlett lag window with the truncation lag increased
four quarters beyond that of (1) and (2).
A RMSE ratio below unity indicates that the SPF consensus (median) forecast has a root-mean-square error
lower than that of the benchmark. This means the SPF is more accurate. We now describe the benchmark models.
The first is perhaps the simplest of all possible benchmarks: A no-change model. In this model, the forecast
for quarter T, the one-step-ahead or current-quarter forecast, is simply the historical value for the prior
quarter (T - 1). There is, in other words, no change in the forecast compared with the historical value.
Moreover, the forecast for the remaining quarters of the horizon is the same as the forecast for the current
quarter. We denote the relative RMSE ratio for this benchmark as RMSE(S/NC), using NC to indicate no change.
The second and third benchmark models generate projections using one or more historical observations of the
the variable forecast, weighted by coefficients estimated from the data. Such autoregressive (AR)
models can be formulated in two ways. We can estimate one model to generate the forecasts at all horizons,
using an iteration method to generate the projections beyond the current quarter (IAR), or we can directly
estimate a new model for each forecast horizon (DAR). The latter formulation has been shown to reduce the
bias in a forecast when the underlying model is characterized by certain types of misspecification. The
root-mean-square error ratios are denoted RMSE(S/IAR) and RMSE(S/DAR), respectively.
The one- through five-step-ahead projections of the benchmark models use information on the quarterly
average of the variable forecast. The latest historical observation is for the quarter that is one quarter
before the quarter of the first projection in the horizon. In contrast, the panelists generate their
projections with the help of additional information. They submit their projections near the middle of each
quarter and hence have access to some monthly indicators for the first month of each quarter, when those
data are released before the survey deadline. This puts the projections of panelists for some variables
at an advantage relative to the corresponding benchmark projections. Moreover, the panelists may also
examine the very recent historical values of such monthly indicators in forming their projections for
quarterly averages. Such monthly statistical momentum represents an advantage not shared by the benchmark
models, which use only quarterly averages. For survey variables whose observations are reported at a
monthly frequency, such as interest rates, industrial production, housing starts, and unemployment, we
estimate and forecast a fourth benchmark model, the DARM. This model adds recent monthly historical values
to the specification of the DAR model. For the projections for unemployment, nonfarm payroll employment,
and interest rates, we add the values of monthly observations, beginning with that for the first month
of the first quarter of the forecast horizon. These values should be in the information set of the survey
panelists at the time they formed their projections. In contrast, for variables such as housing
starts and industrial production, we include only lagged values of monthly observations. For such
variables, the panelists would not have known the monthly observation for the first month of the first
quarter of the forecast horizon. In general, we find that adding monthly observations to the benchmark
DAR models improves accuracy. Indeed, for the projections for interest rates and the unemployment rate,
the accuracy of the benchmark DARM projections rivals that of the SPF projections.
2.2. Real-Time Data.
All benchmark models are estimated on a rolling, fixed window of 60 real-time quarterly observations.
Lag lengths, based on either the Akaike information criterion (AIC) or the Schwarz information
criterion (SIC), are re-estimated each period. The tables below indicate whether the lag length was
was chosen by the AIC or SIC.
We would like to make the comparison between the SPF forecast and the forecasts of each benchmark as
fair as possible. Therefore, we must subject the benchmark models to the same data environment the
survey panelists faced when they made their projections. This is important because macroeconomic
data are revised often, and we do not want the benchmark models to use a data set that differs from the one
our panelists would have used. We estimate and forecast the benchmark models with real-time data from the
Philadelphia Fed real-time data set, using the vintage of data that the survey panelists would have had
at the time they generated their own projections. (For more information on the Philadelphia Fed
real-time data set, go to www.philadelphiafed.org/econ/forecast/real-time-data/.)
An open question in the literature on forecasting is: What version or vintage of the data should we use to
compute the errors? A closely related question is: What version of the data are professional forecasters
trying to predict? Our computations take no strong position on these questions. In Tables 1A - 1B, we
evaluate the projections (SPF and benchmark) with five alternative measures of the historical values, all
from the Philadelphia Fed real-time data set. These measures range from the initial-release values to the
values as we know them today. All together, we compute the forecast error statistics using the following
five alternative measures of historical values:
(1) The initial or first-release value;
(2) The revised value as it appears one quarter after the initial release;
(3) The revised value as it appears five quarters after the initial release;
(4) The revised value as it appears nine quarters after the initial release;
(5) The revised value as it appears today.
Each measure of the historical value has advantages and disadvantages. The initial-release value is the
first measure released by government statistical agencies. A forecaster might be very interested in this
measure because it enables him to evaluate his latest forecast soon after he generated it. However, early
releases of the data are often subject to large measurement error. Subsequent releases [(2) - (5)]
are more accurate, but they are available much later than the initial release. As we go from the first
measure to the fifth, we get more reliability, at the cost of higher delays in availability.
The last two columns in Table 1A report the number of observations that we used to compute the error
statistics. Some observations are omitted because the data are missing in the real-time data set,
such as occurred when federal government statistical agencies closed in late 1995.
2.3. Recent Projections and Realizations.
Tables 2 to 7 provide information on recent projections and realizations. They show how we align the data
prior to computing the forecast errors that form the backbone of the computations in Tables 1A - 1B. Any
error can be written as the equation given by error = realization - forecast. For our computations, we must
be more precise because, for each projection (SPF and benchmarks), we have different periods forecast (T)
different forecast horizons (h), and several measures of the realization (m). Thus, we can define the
forecast error more precisely as
error( T, h, m ) = realization( T, m ) - forecast( T, h ).
Tables 2 to 7 are organized along these lines. Table 2 shows recent forecasts from the SPF. Each column
gives the projection for a different horizon or forecast step (h), beginning with that for the current
quarter, defined as the quarter in which we conducted the survey. The dates (T) given in the rows show the
periods forecast. These also correspond to the dates that we conducted the survey. Tables 3 to 6 report the
recent projections of the four benchmark models. They are organized in the same way as Table 2. Table 7
reports recent values of the five alternative realizations (m) we use to compute the error statistics.
2.4. Qualifications.
We note two minor qualifications to the methods discussed above. The first concerns the vintage of data
that we used to estimate and forecast the benchmark models for CPI inflation. The second concerns the five
measures of realizations used for the unemployment rate, nonfarm payroll employment, and CPI inflation. To
estimate and forecast the benchmark models for CPI inflation, we use the vintage of data that would have
been available in the middle of each quarter. This postdates by one month the vintage that SPF panelists
would have had at their disposal when they formed their projections.
To compute the realizations for unemployment, nonfarm payroll employment, and CPI inflation, we use the
vintages associated with the middle of each quarter. The measure that we call initial comes from this
vintage, even though the initial estimate was available in the vintage dated one month earlier. Thus,
for these variables, our initial estimate reflects some revision by government statistical agencies.
The effect for unemployment and CPI inflation is likely small. The effect could be somewhat larger for
nonfarm payroll employment.
3. DESCRIPTION OF GRAPHS.
3.1. Root-Mean-Square Errors.
For each sample period shown in Table 1, we provide graphs of the root-mean-square error for the SPF forecast.
There is one page for each sample period. On each page, we plot (for each forecast horizon) the RMSE
on the y-axis. The x-axis shows the measure of the historical value that we used to compute the RMSE.
These range from the value on its initial release to the value one quarter later to the value as we know it
now (at the time we made the computation).
The graphs provide a tremendous amount of information. If we focus on a particular graph, we can see how
a change in the measure of the realization (x-axis) affects the root-mean-square-error measure of accuracy.
The effect is pronounced for some variables, such as real GDP and some of its components. For others,
there is little or no effect. For example, because the historical data on interest rates are not revised,
the estimated RMSE is the same in each case.
If we compare a particular point on one graph with the same point on another, we see how the forecast
horizon affects accuracy. In general, the RMSE rises (accuracy falls) as the forecast horizon lengthens.
Finally, if we compare a graph on one page with the corresponding graph on another page, we see how our
estimates of accuracy in the SPF change with the sample period. Periods characterized by a high degree of
economic turbulence will generally produce large RMSEs.
3.2. Fan Charts.
The last chart plots recent historical values and the latest SPF forecast. It also shows confidence
intervals for the forecast, based on back-of-the-envelope calculations. The historical values and
the SPF forecast are those associated with the latest vintage of data and survey, respectively,
available at the time we ran our computer programs. The confidence intervals are constructed under the
assumption that the historical forecast errors over the sample (shown in the footnote) follow a normal
distribution with a mean of zero and a variance given by the squared root-mean-square error. The latter
is estimated over the aforementioned sample, using the measure of history listed in the footnote.
_________________________________________________________________________________________________
Table 1A.
Forecast Error Summary Statistics for SPF Variable: EMP (Nonfarm Payroll Employment)
_________________________________________________________________________________________________
Computed Over Various Sample Periods
Various Measures of Realizations
Transformation: Q/Q Growth Rate
Lag Length for IAR(p), DAR(p), and DARM(p) Models: AIC
Source for Historical Realizations: Bureau of Labor Statistics via Haver Analytics
Last Updated: 08/29/2021 15:45
_________________________________________________________________________________________________
H ME(S) MAE(S) RMSE(S) RMSE(S/NC) RMSE(S/IAR) RMSE(S/DAR) RMSE(S/DARM) Nspf N
History: Initial Release
2004:04-2019:01
1 -0.01 0.26 0.38 0.45 0.43 0.43 0.87 58 58
2 -0.15 0.51 0.86 0.68 0.63 0.60 0.71 58 58
3 -0.26 0.59 1.14 0.73 0.66 0.68 0.75 58 58
4 -0.38 0.69 1.39 0.77 0.69 0.75 0.83 58 58
5 -0.41 0.72 1.47 0.72 0.64 0.77 0.80 58 58
H ME(S) MAE(S) RMSE(S) RMSE(S/NC) RMSE(S/IAR) RMSE(S/DAR) RMSE(S/DARM) Nspf N
History: One Qtr After Initial Release
2004:04-2019:01
1 -0.01 0.28 0.41 0.49 0.47 0.47 0.92 58 58
2 -0.15 0.54 0.88 0.70 0.65 0.61 0.73 58 58
3 -0.26 0.62 1.17 0.75 0.67 0.68 0.76 58 58
4 -0.39 0.71 1.42 0.78 0.70 0.75 0.84 58 58
5 -0.41 0.76 1.50 0.73 0.65 0.77 0.80 58 58
H ME(S) MAE(S) RMSE(S) RMSE(S/NC) RMSE(S/IAR) RMSE(S/DAR) RMSE(S/DARM) Nspf N
History: Five Qtrs After Initial Release
2004:04-2019:01
1 -0.05 0.38 0.61 0.65 0.61 0.61 0.96 58 58
2 -0.19 0.57 1.05 0.77 0.72 0.69 0.79 58 58
3 -0.30 0.67 1.34 0.80 0.73 0.74 0.81 58 58
4 -0.43 0.77 1.59 0.82 0.75 0.79 0.87 58 58
5 -0.45 0.82 1.67 0.77 0.69 0.80 0.83 58 58
H ME(S) MAE(S) RMSE(S) RMSE(S/NC) RMSE(S/IAR) RMSE(S/DAR) RMSE(S/DARM) Nspf N
History: Nine Qtrs After Initial Release
2004:04-2019:01
1 -0.06 0.38 0.61 0.65 0.62 0.62 0.99 58 58
2 -0.20 0.58 1.06 0.77 0.72 0.69 0.80 58 58
3 -0.31 0.69 1.36 0.80 0.73 0.73 0.81 58 58
4 -0.44 0.79 1.62 0.83 0.75 0.80 0.87 58 58
5 -0.46 0.83 1.69 0.78 0.70 0.80 0.83 58 58
H ME(S) MAE(S) RMSE(S) RMSE(S/NC) RMSE(S/IAR) RMSE(S/DAR) RMSE(S/DARM) Nspf N
History: Latest Vintage
2004:04-2019:01
1 -0.06 0.37 0.60 0.64 0.61 0.61 0.99 58 58
2 -0.20 0.57 1.05 0.77 0.72 0.69 0.79 58 58
3 -0.30 0.69 1.36 0.80 0.73 0.73 0.81 58 58
4 -0.43 0.78 1.61 0.83 0.76 0.80 0.87 58 58
5 -0.45 0.82 1.68 0.78 0.69 0.80 0.83 58 58
Notes for Table 1A.
(1) The forecast horizon is given by H, where H = 1 is the SPF forecast for the current quarter.
(2) The headers ME(S), MAE(S), and RMSE(S) are mean error, mean absolute error, and
root-mean-square error for the SPF.
(3) The header RMSE(S/NC) is the ratio of the SPF RMSE to that of the no-change (NC) model.
(4) The headers RMSE(S/IAR), RMSE(S/DAR) and RMSE(S/DARM) are the ratios of the SPF RMSE to the RMSE
of the iterated and direct autoregressive models and the direct autoregressive model augmented
with monthly observations, respectively. All models are estimated on a rolling window of 60
observations from the Philadelphia Fed real-time data set.
(5) The headers Nspf and N are the number of observations analyzed for the SPF and benchmark models.
(6) When the variable forecast is a growth rate or an interest rate, it is expressed in annualized
percentage points. When the variable forecast is the unemployment rate, it is expressed in percentage
points.
(7) Sample periods refer to the dates forecast, not the dates when the forecasts were made.
Source: Tom Stark, Research Department, FRB Philadelphia.
________________________________________________________________________________________________________
Table 1B.
Ratios of Root-Mean-Square Errors for SPF Variable: EMP (Nonfarm Payroll Employment)
Alternative P-Values in Parentheses
________________________________________________________________________________________________________
Computed Over Various Sample Periods
Various Measures of Realizations
Transformation: Q/Q Growth Rate
Lag Length for IAR(p), DAR(p), and DARM(p) Models: AIC
Source for Historical Realizations: Bureau of Labor Statistics via Haver Analytics
Last Updated: 08/29/2021 15:45
________________________________________________________________________________________________________
History: Initial Release
2004:04-2019:01
H RMSE(S/NC) RMSE(S/IAR) RMSE(S/DAR) RMSE(S/DARM) N1 N2 N3 N4
1 0.449 0.430 0.430 0.866 58 58 58 58
(0.000) (0.003) (0.003) (0.031)
(0.001) (0.005) (0.005) (0.037)
(0.047) (0.032) (0.032) (0.066)
2 0.677 0.633 0.596 0.712 58 58 58 58
(0.066) (0.023) (0.031) (0.012)
(0.079) (0.030) (0.040) (0.017)
(0.145) (0.075) (0.081) (0.058)
3 0.735 0.662 0.676 0.748 58 58 58 58
(0.294) (0.155) (0.085) (0.109)
(0.320) (0.179) (0.104) (0.131)
(0.230) (0.129) (0.080) (0.084)
4 0.769 0.689 0.750 0.834 58 58 58 58
(0.343) (0.178) (0.071) (0.071)
(0.377) (0.211) (0.096) (0.095)
(0.278) (0.158) (0.056) (0.066)
5 0.723 0.643 0.768 0.800 58 58 58 58
(0.228) (0.126) (0.009) (0.016)
(0.271) (0.164) (0.019) (0.030)
(0.209) (0.139) (0.014) (0.017)
History: One Qtr After Initial Release
2004:04-2019:01
H RMSE(S/NC) RMSE(S/IAR) RMSE(S/DAR) RMSE(S/DARM) N1 N2 N3 N4
1 0.491 0.469 0.469 0.921 58 58 58 58
(0.001) (0.004) (0.004) (0.173)
(0.001) (0.006) (0.006) (0.182)
(0.043) (0.028) (0.028) (0.129)
2 0.699 0.653 0.614 0.731 58 58 58 58
(0.073) (0.019) (0.027) (0.009)
(0.087) (0.026) (0.035) (0.014)
(0.149) (0.067) (0.075) (0.049)
3 0.745 0.669 0.683 0.757 58 58 58 58
(0.299) (0.150) (0.080) (0.107)
(0.324) (0.174) (0.100) (0.129)
(0.233) (0.125) (0.076) (0.078)
4 0.777 0.696 0.754 0.837 58 58 58 58
(0.345) (0.172) (0.065) (0.058)
(0.378) (0.204) (0.088) (0.080)
(0.280) (0.153) (0.051) (0.056)
5 0.733 0.651 0.772 0.803 58 58 58 58
(0.228) (0.121) (0.007) (0.013)
(0.271) (0.158) (0.016) (0.026)
(0.211) (0.135) (0.012) (0.015)
History: Five Qtrs After Initial Release
2004:04-2019:01
H RMSE(S/NC) RMSE(S/IAR) RMSE(S/DAR) RMSE(S/DARM) N1 N2 N3 N4
1 0.647 0.614 0.614 0.961 58 58 58 58
(0.002) (0.006) (0.006) (0.170)
(0.004) (0.009) (0.009) (0.179)
(0.038) (0.029) (0.029) (0.169)
2 0.770 0.721 0.687 0.792 58 58 58 58
(0.038) (0.008) (0.015) (0.001)
(0.048) (0.012) (0.021) (0.002)
(0.116) (0.046) (0.055) (0.021)
3 0.805 0.729 0.737 0.809 58 58 58 58
(0.284) (0.117) (0.046) (0.068)
(0.309) (0.139) (0.061) (0.086)
(0.216) (0.095) (0.048) (0.048)
4 0.822 0.746 0.789 0.870 58 58 58 58
(0.351) (0.147) (0.040) (0.030)
(0.385) (0.178) (0.058) (0.046)
(0.279) (0.130) (0.030) (0.033)
5 0.772 0.692 0.797 0.828 58 58 58 58
(0.229) (0.108) (0.003) (0.004)
(0.272) (0.144) (0.009) (0.011)
(0.210) (0.125) (0.007) (0.007)
History: Nine Qtrs After Initial Release
2004:04-2019:01
H RMSE(S/NC) RMSE(S/IAR) RMSE(S/DAR) RMSE(S/DARM) N1 N2 N3 N4
1 0.653 0.623 0.623 0.987 58 58 58 58
(0.004) (0.009) (0.009) (0.716)
(0.006) (0.012) (0.012) (0.720)
(0.037) (0.029) (0.029) (0.730)
2 0.772 0.723 0.691 0.800 58 58 58 58
(0.037) (0.007) (0.016) (0.001)
(0.046) (0.011) (0.022) (0.002)
(0.113) (0.044) (0.057) (0.023)
3 0.801 0.726 0.734 0.814 58 58 58 58
(0.262) (0.108) (0.040) (0.075)
(0.288) (0.130) (0.054) (0.094)
(0.199) (0.089) (0.044) (0.053)
4 0.831 0.755 0.796 0.869 58 58 58 58
(0.368) (0.156) (0.041) (0.025)
(0.402) (0.188) (0.060) (0.040)
(0.293) (0.136) (0.032) (0.031)
5 0.776 0.697 0.800 0.831 58 58 58 58
(0.228) (0.110) (0.004) (0.004)
(0.271) (0.146) (0.011) (0.011)
(0.210) (0.127) (0.008) (0.007)
History: Latest Vintage
2004:04-2019:01
H RMSE(S/NC) RMSE(S/IAR) RMSE(S/DAR) RMSE(S/DARM) N1 N2 N3 N4
1 0.642 0.613 0.613 0.991 58 58 58 58
(0.003) (0.007) (0.007) (0.794)
(0.005) (0.010) (0.010) (0.797)
(0.033) (0.026) (0.026) (0.793)
2 0.769 0.719 0.685 0.793 58 58 58 58
(0.033) (0.007) (0.016) (0.001)
(0.043) (0.011) (0.022) (0.002)
(0.112) (0.045) (0.056) (0.020)
3 0.802 0.727 0.734 0.814 58 58 58 58
(0.259) (0.106) (0.037) (0.069)
(0.285) (0.128) (0.051) (0.087)
(0.198) (0.089) (0.044) (0.048)
4 0.834 0.757 0.797 0.869 58 58 58 58
(0.376) (0.158) (0.040) (0.023)
(0.409) (0.190) (0.059) (0.036)
(0.297) (0.136) (0.030) (0.029)
5 0.775 0.695 0.798 0.831 58 58 58 58
(0.228) (0.109) (0.004) (0.004)
(0.271) (0.145) (0.011) (0.010)
(0.211) (0.126) (0.008) (0.006)
Notes for Table 1B.
(1) The forecast horizon is given by H, where H = 1 is the SPF forecast for the current quarter.
(2) The headers RMSE(S/NC), RMSE(S/IAR), RMSE(S/DAR), and RMSE(S/DARM) are the ratios of the SPF
root-mean-square error to that of the benchmark models: No-change (NC), indirect autoregression (IAR),
direct autoregession (DAR), and direct autoregression augmented with monthly information (DARM).
These statistics may differ slightly from those reported in Table 1A because they incorporate only
those observations common to both the SPF and the benchmark model. The previous statistics make use
of all available observations for each model.
(3) All models are estimated on a rolling window of 60 observations from the Philadelphia Fed real-time
data set.
(4) A set of three two-sided p-values (in parentheses) accompanies each statistic. These are the p-values
for the test of the equality of mean-square-error. The first is for the Diebold-Mariano (1995, JBES)
statistic, using a uniform lag window with the trunction lag set to the forecast horizon minus one.
(The tables report the p-values using a Bartlett window when the uniform window produces a negative
standard error.) The second is for the Harvey-Leybourne-Newbold (1997, IJF) correction to the
Diebold-Mariano statistic. The third is for the Diebold-Mariano statistic, using a Bartlett lag
window with the truncation lag increased four quarters.
(5) The headers N1, N2, N3, and N4 show the number of observations used in constructing each ratio of
root-mean-square errors.
(6) Sample periods refer to the dates forecast, not the dates when the forecasts were made.
Source: Tom Stark, Research Department, FRB Philadelphia.
______________________________________________________________________________
Table 2. Recent SPF Forecasts
(Dated at the Quarter Forecast)
______________________________________________________________________________
Variable: EMP (Nonfarm Payroll Employment)
By Forecast Step (1 to 5)
Transformation: Q/Q Growth Rate
Last Updated: 08/29/2021 15:45
______________________________________________________________________________
Qtr Forecast Step 1 Step 2 Step 3 Step 4 Step 5
2014:04 1.924 1.834 1.719 1.876 1.763
2015:01 2.327 1.824 1.800 1.724 1.745
2015:02 1.673 2.007 1.679 1.799 1.764
2015:03 1.900 1.906 1.895 1.781 1.718
2015:04 1.711 1.872 1.894 1.950 1.716
2016:01 1.647 1.590 1.563 1.494 1.807
2016:02 1.596 1.541 1.629 1.610 1.500
2016:03 1.540 1.607 1.641 1.610 1.588
2016:04 1.476 1.418 1.468 1.271 1.513
2017:01 1.532 1.338 1.329 1.509 1.472
2017:02 1.468 1.383 1.485 1.367 1.378
2017:03 1.538 1.408 1.394 1.371 1.317
2017:04 1.506 1.323 1.358 1.318 1.366
2018:01 1.503 1.352 1.274 1.366 1.291
2018:02 1.475 1.397 1.364 1.327 1.300
2018:03 1.600 1.420 1.373 1.278 1.346
2018:04 1.642 1.400 1.297 1.230 1.262
2019:01 1.600 1.387 1.300 1.297 1.398
2019:02 1.526 1.255 1.348 1.300 1.218
2019:03 1.201 1.313 1.345 1.275 1.200
2019:04 1.296 1.178 1.230 1.130 1.137
2020:01 1.412 1.055 1.090 1.055 1.138
2020:02 -48.012 1.332 1.260 1.152 1.218
2020:03 19.899 23.463 1.044 0.964 0.853
2020:04 6.011 3.462 8.182 0.915 1.000
2021:01 1.210 4.021 3.127 4.526 0.895
2021:02 4.862 3.365 3.572 1.703 6.470
2021:03 5.876 6.376 3.761 3.715 2.425
2021:04 NA 4.215 3.987 4.747 3.305
2022:01 NA NA 3.839 3.037 3.646
2022:02 NA NA NA 3.277 2.317
2022:03 NA NA NA NA 2.119
Notes for Table 2.
(1) Each column gives the sequence of SPF projections for a given forecast step. The forecast steps
range from one (the forecast for the quarter in which the survey was conducted) to four quarters
in the future (step 5).
(2) The dates listed in the rows are the dates forecast, not the dates when the forecasts were made,
with the exception of the forecast at step one, for which the two dates coincide.
Source: Tom Stark, Research Department, FRB Philadelphia.
______________________________________________________________________________________________
Table 3. Recent Benchmark Model 1 IAR Forecasts
(Dated at the Quarter Forecast)
______________________________________________________________________________________________
Variable: EMP (Nonfarm Payroll Employment)
By Forecast Step (1 to 5)
Transformation: Q/Q Growth Rate
Lag Length for IAR(p): AIC
Source for Historical Realizations: Bureau of Labor Statistics via Haver Analytics
Last Updated: 08/29/2021 15:45
______________________________________________________________________________________________
Qtr Forecast Step 1 Step 2 Step 3 Step 4 Step 5
2014:04 1.945 1.806 0.828 0.947 0.484
2015:01 2.088 1.620 1.466 0.663 0.773
2015:02 1.909 1.792 1.323 1.199 0.553
2015:03 1.546 1.530 1.405 1.043 0.944
2015:04 1.619 1.388 1.201 1.097 0.826
2016:01 1.866 1.493 1.253 0.924 0.839
2016:02 1.655 1.717 1.382 1.137 0.721
2016:03 1.049 1.389 1.586 1.285 1.037
2016:04 1.491 0.812 1.150 1.472 1.200
2017:01 1.365 1.445 0.700 0.949 1.371
2017:02 1.316 1.152 1.263 0.627 0.796
2017:03 1.263 1.229 1.064 1.144 0.604
2017:04 1.168 1.137 1.104 0.974 1.022
2018:01 1.453 1.083 1.044 1.024 0.928
2018:02 1.605 1.344 0.999 0.962 0.956
2018:03 1.686 1.526 1.235 0.939 0.903
2018:04 1.556 1.567 1.392 1.133 0.892
2019:01 1.578 1.426 1.436 1.281 1.049
2019:02 1.539 1.433 1.296 1.316 1.178
2019:03 1.045 1.405 1.289 1.188 1.216
2019:04 1.205 0.881 1.274 1.175 1.102
2020:01 1.622 1.195 0.822 1.168 1.084
2020:02 0.676 1.542 1.121 0.778 1.085
2020:03 -85.045 0.499 1.426 1.085 0.769
2020:04 -51.992 -128.728 0.514 1.309 1.046
2021:01 -52.572 48.770 -163.533 0.535 1.204
2021:02 0.225 45.103 -76.992 -181.419 0.594
2021:03 -0.440 0.713 3.867 87.306 -174.969
2021:04 NA 0.940 0.584 -72.599 -121.418
2022:01 NA NA 0.577 0.618 77.139
2022:02 NA NA NA 0.673 0.609
2022:03 NA NA NA NA 0.648
Notes for Table 3.
(1) Each column gives the sequence of benchmark IAR projections for a given forecast step. The forecast
steps range from one to five. The first step corresponds to the forecast that SPF panelists
make for the quarter in which the survey is conducted.
(2) The dates listed in the rows are the dates forecast, not the dates when the forecasts were made,
with the exception of the forecast at step one, for which the two dates coincide.
(3) The IAR benchmark model is estimated on a fixed 60-quarter rolling window. Its forecasts are
computed with the indirect method. Estimation uses data from the Philadelphia Fed real-time
data set.
Source: Tom Stark, Research Department, FRB Philadelphia.
_________________________________________________________________________________________________
Table 4. Recent Benchmark Model 2 No-Change Forecasts
(Dated at the Quarter Forecast)
_________________________________________________________________________________________________
Variable: EMP (Nonfarm Payroll Employment)
By Forecast Step (1 to 5)
Transformation: Q/Q Growth Rate
Source for Historical Realizations: Bureau of Labor Statistics via Haver Analytics
Last Updated: 08/29/2021 15:45
_________________________________________________________________________________________________
Qtr Forecast Step 1 Step 2 Step 3 Step 4 Step 5
2014:04 2.090 2.241 1.486 1.832 1.409
2015:01 2.463 2.090 2.241 1.486 1.832
2015:02 2.206 2.463 2.090 2.241 1.486
2015:03 1.728 2.206 2.463 2.090 2.241
2015:04 1.765 1.728 2.206 2.463 2.090
2016:01 2.037 1.765 1.728 2.206 2.463
2016:02 1.880 2.037 1.765 1.728 2.206
2016:03 1.285 1.880 2.037 1.765 1.728
2016:04 1.738 1.285 1.880 2.037 1.765
2017:01 1.419 1.738 1.285 1.880 2.037
2017:02 1.538 1.419 1.738 1.285 1.880
2017:03 1.356 1.538 1.419 1.738 1.285
2017:04 1.295 1.356 1.538 1.419 1.738
2018:01 1.510 1.295 1.356 1.538 1.419
2018:02 1.708 1.510 1.295 1.356 1.538
2018:03 1.761 1.708 1.510 1.295 1.356
2018:04 1.686 1.761 1.708 1.510 1.295
2019:01 1.745 1.686 1.761 1.708 1.510
2019:02 1.687 1.745 1.686 1.761 1.708
2019:03 1.179 1.687 1.745 1.686 1.761
2019:04 1.360 1.179 1.687 1.745 1.686
2020:01 1.639 1.360 1.179 1.687 1.745
2020:02 0.758 1.639 1.360 1.179 1.687
2020:03 -39.985 0.758 1.639 1.360 1.179
2020:04 22.908 -39.985 0.758 1.639 1.360
2021:01 5.167 22.908 -39.985 0.758 1.639
2021:02 2.066 5.167 22.908 -39.985 0.758
2021:03 4.808 2.066 5.167 22.908 -39.985
2021:04 NA 4.808 2.066 5.167 22.908
2022:01 NA NA 4.808 2.066 5.167
2022:02 NA NA NA 4.808 2.066
2022:03 NA NA NA NA 4.808
Notes for Table 4.
(1) Each column gives the sequence of benchmark no-change projections for a given forecast step.
The forecast steps range from one to five. The first step corresponds to the forecast that SPF
panelists make for the quarter in which the survey is conducted.
(2) The dates listed in the rows are the dates forecast, not the dates when the forecasts were made,
with the exception of the forecast at step one, for which the two dates coincide.
(3) The projections use data from the Philadelphia Fed real-time data set.
Source: Tom Stark, Research Department, FRB Philadelphia.
______________________________________________________________________________________________
Table 5. Recent Benchmark Model 3 DAR Forecasts
(Dated at the Quarter Forecast)
______________________________________________________________________________________________
Variable: EMP (Nonfarm Payroll Employment)
By Forecast Step (1 to 5)
Transformation: Q/Q Growth Rate
Lag Length for DAR(p): AIC
Source for Historical Realizations: Bureau of Labor Statistics via Haver Analytics
Last Updated: 08/29/2021 15:45
______________________________________________________________________________________________
Qtr Forecast Step 1 Step 2 Step 3 Step 4 Step 5
2014:04 1.945 1.742 0.817 1.012 0.545
2015:01 2.088 1.672 1.667 0.619 0.583
2015:02 1.909 1.758 1.159 1.320 0.472
2015:03 1.546 1.532 1.543 0.878 0.817
2015:04 1.619 0.954 1.141 1.167 0.778
2016:01 1.866 1.116 0.780 0.847 0.842
2016:02 1.655 1.562 1.097 0.578 0.789
2016:03 1.049 1.385 1.305 0.854 0.688
2016:04 1.491 0.775 1.139 1.010 0.722
2017:01 1.365 1.391 0.700 0.875 0.793
2017:02 1.316 1.177 1.426 0.594 0.806
2017:03 1.263 1.182 0.896 1.203 0.748
2017:04 1.168 1.136 1.250 0.780 0.925
2018:01 1.453 1.064 1.036 1.116 0.918
2018:02 1.605 1.340 0.983 0.964 0.997
2018:03 1.686 1.504 1.193 0.985 0.976
2018:04 1.556 1.563 1.403 1.087 0.973
2019:01 1.578 1.425 1.387 1.246 1.050
2019:02 1.539 1.423 1.333 1.237 1.114
2019:03 1.045 1.389 1.354 1.197 1.148
2019:04 1.205 0.885 1.314 1.223 1.149
2020:01 1.622 1.125 0.861 1.196 1.176
2020:02 0.676 1.567 1.236 0.815 1.167
2020:03 -85.045 0.531 1.360 1.167 1.020
2020:04 -51.992 -23.621 0.453 1.243 1.063
2021:01 -52.572 11.842 -18.603 0.473 1.110
2021:02 0.225 0.388 12.362 -14.302 0.869
2021:03 -0.440 0.576 2.871 9.691 -7.271
2021:04 NA 0.606 0.653 2.447 4.863
2022:01 NA NA 0.910 1.100 1.562
2022:02 NA NA NA 0.446 0.862
2022:03 NA NA NA NA 1.470
Notes for Table 5.
(1) Each column gives the sequence of benchmark DAR projections for a given forecast step. The forecast
steps range from one to five. The first step corresponds to the forecast that SPF panelists
make for the quarter in which the survey is conducted.
(2) The dates listed in the rows are the dates forecast, not the dates when the forecasts were made,
with the exception of the forecast at step one, for which the two dates coincide.
(3) The DAR benchmark model is estimated on a fixed 60-quarter rolling window. Its forecasts are
computed with the direct method. Estimation uses data from the Philadelphia Fed real-time
data set.
Source: Tom Stark, Research Department, FRB Philadelphia.
______________________________________________________________________________________________
Table 6. Recent Benchmark Model 4 DARM Forecasts
(Dated at the Quarter Forecast)
______________________________________________________________________________________________
Variable: EMP (Nonfarm Payroll Employment)
By Forecast Step (1 to 5)
Transformation: Q/Q Growth Rate
Lag Length for DARM(p): AIC
Source for Historical Realizations: Bureau of Labor Statistics via Haver Analytics
Last Updated: 08/29/2021 15:45
______________________________________________________________________________________________
Qtr Forecast Step 1 Step 2 Step 3 Step 4 Step 5
2014:04 1.870 1.659 1.457 0.879 0.862
2015:01 2.591 1.390 1.392 1.444 0.859
2015:02 1.475 2.005 0.987 0.884 1.085
2015:03 1.908 1.660 1.127 0.759 0.557
2015:04 1.701 1.595 1.265 1.030 0.509
2016:01 1.717 1.617 0.766 1.320 0.824
2016:02 1.535 1.338 1.693 1.056 1.253
2016:03 1.905 1.186 0.827 1.381 0.666
2016:04 1.395 1.184 0.841 0.697 1.231
2017:01 1.562 1.343 1.231 0.764 0.368
2017:02 1.362 1.318 1.211 0.679 0.589
2017:03 1.656 1.515 1.164 0.748 0.688
2017:04 1.373 1.304 1.394 1.120 0.696
2018:01 1.502 1.595 1.204 1.602 1.195
2018:02 1.439 1.448 1.615 0.916 1.471
2018:03 1.629 1.607 1.263 1.997 0.862
2018:04 1.717 1.341 1.219 1.257 1.906
2019:01 2.039 1.942 1.024 1.395 1.187
2019:02 1.546 1.858 1.646 0.900 1.210
2019:03 1.185 1.399 1.788 1.895 0.763
2019:04 1.251 0.850 1.598 1.639 1.655
2020:01 1.607 1.236 0.817 1.194 1.526
2020:02 -43.586 1.670 0.949 0.555 1.342
2020:03 37.102 -41.703 1.524 0.980 0.798
2020:04 12.233 19.151 -54.355 1.556 0.806
2021:01 1.188 5.407 -21.151 -49.493 1.392
2021:02 8.217 0.871 6.562 26.238 -44.710
2021:03 6.133 2.370 4.421 -0.872 114.209
2021:04 NA 1.492 1.879 -2.158 -13.828
2022:01 NA NA 1.633 5.164 -4.318
2022:02 NA NA NA 1.772 6.106
2022:03 NA NA NA NA 3.390
Notes for Table 6.
(1) Each column gives the sequence of benchmark DARM projections for a given forecast step. The forecast
steps range from one to five. The first step corresponds to the forecast that SPF panelists
make for the quarter in which the survey is conducted.
(2) The dates listed in the rows are the dates forecast, not the dates when the forecasts were made,
with the exception of the forecast at step one, for which the two dates coincide.
(3) The DARM benchmark model is estimated on a fixed 60-quarter rolling window. Its forecasts are
computed with the direct method and incorporate recent monthly values of the dependent variable.
Estimation uses data from the Philadelphia Fed real-time data set.
Source: Tom Stark, Research Department, FRB Philadelphia.
______________________________________________________________________
Table 7. Recent Realizations (Various Measures)
Philadelphia Fed Real-Time Data Set
______________________________________________________________________
Variable: EMP (Nonfarm Payroll Employment)
Transformation: Q/Q Growth Rate
Source for Historical Realizations: Bureau of Labor Statistics via Haver Analytics
Last Updated: 08/29/2021 15:45
Column (1): Initial Release
Column (2): One Qtr After Initial Release
Column (3): Five Qtrs After Initial Release
Column (4): Nine Qtrs After Initial Release
Column (5): Latest Vintage
_______________________________________________________________________
Obs. Date (1) (2) (3) (4) (5)
2014:04 2.463 2.463 2.262 2.268 2.329
2015:01 2.206 2.239 2.103 2.000 1.930
2015:02 1.728 1.742 1.869 1.994 2.005
2015:03 1.765 1.907 1.833 1.795 1.889
2015:04 2.037 2.045 1.987 2.037 1.985
2016:01 1.880 1.859 1.716 1.681 1.661
2016:02 1.285 1.266 1.429 1.497 1.512
2016:03 1.738 1.970 2.046 1.984 2.095
2016:04 1.419 1.417 1.526 1.495 1.435
2017:01 1.538 1.511 1.646 1.630 1.527
2017:02 1.356 1.336 1.304 1.396 1.484
2017:03 1.295 1.520 1.457 1.483 1.530
2017:04 1.510 1.524 1.476 1.462 1.453
2018:01 1.708 1.727 1.751 1.671 1.591
2018:02 1.761 1.725 1.890 1.763 1.797
2018:03 1.686 1.805 1.526 1.541 1.541
2018:04 1.745 1.750 1.308 1.346 1.346
2019:01 1.687 1.655 1.301 1.209 1.209
2019:02 1.179 1.166 1.139 NA 1.109
2019:03 1.360 1.471 1.427 NA 1.427
2019:04 1.639 1.672 1.643 NA 1.643
2020:01 0.758 0.352 0.348 NA 0.348
2020:02 -39.985 -39.991 NA NA -39.989
2020:03 22.908 23.333 NA NA 23.333
2020:04 5.167 5.089 NA NA 5.089
2021:01 2.066 2.080 NA NA 2.080
2021:02 4.808 NA NA NA 4.808
2021:03 NA NA NA NA NA
2021:04 NA NA NA NA NA
2022:01 NA NA NA NA NA
2022:02 NA NA NA NA NA
2022:03 NA NA NA NA NA
Notes for Table 7.
(1) Each column reports a sequence of realizations from the Philadelphia Fed real-time data set.
(2) The date listed in each row is the observation date.
(3) Moving across a particular row shows how the observation is revised in subsequent releases.
Source: Tom Stark, Research Department, FRB Philadelphia.