By Scott Irwin and Darrel Good
Department of Agricultural and Consumer Economics
University of Illinois
As the calendar turns to May, the market will begin focusing in earnest on the yield prospects for the 2015 U.S. corn crop (farmdoc daily, April 27, 2015). One of the key early assessments is the projection of the U.S. average corn yield by the USDA's World Agricultural Outlook Board (WAOB). Since 1993, those projections have been used in the May, June, and July WASDE reports to make supply, ending stocks, and price projections for the upcoming marketing year. The May 2015 WASDE report will be released on May 12. Previous research has shown that the release of these early projections--particularly in May--has a significant impact on prices in the corn futures market (Isengildina-Massa et al., 2008). While it is clear that the WAOB yield forecasts are perceived by market participants as containing important new information, these forecasts appear to be poorly understood by many and often confused with later forecasts released by the National Agricultural Statistics Service (NASS) of the USDA. The purpose of today's article is to describe the methodology for WAOB corn yield forecasts, demonstrate how the methodology has changed over time, and evaluate the historical accuracy of the forecasts. The analysis of forecast accuracy updates results published in an earlier research report on USDA forecasts and estimates (Irwin, Sanders, and Good, 2014). That evaluation was for the period 1993 through 2012. Here we extend the analysis for corn through the 2014 crop year.
WAOB Forecasting Procedures
To begin, it is important to understand that WAOB and NASS corn yield forecasts are based on entirely different procedures. WAOB forecasts of national corn yield are based on relatively simple trend analysis of historical yields, sometimes modified by planting progress and/or weather conditions. The WAOB corn yield forecasts are released as part of the May, June, and July WASDE reports each year. NASS forecasts of state and national corn yield are based on large scale farmer surveys and field measurement surveys. These forecasts are released in monthly crop production reports from August through November each year, with the final yield estimates released in January after harvest. The NASS corn yield forecasts are released simultaneously with WASDE reports and the forecasts are used without adjustment in WASDE supply projections. We have described the procedures used in making NASS forecasts and provided an evaluation of the accuracy of those forecasts in earlier research reports (Good and Irwin, 2011; Irwin, Sanders, and Good, 2014) and several farmdoc daily articles (August 19, 2011; September 1, 2011; August 28, 2013; August 29, 2014).
With that background, we can turn our attention to describing the methodology for WAOB corn yield forecasts in more detail and how the methodology has changed over time. The procedure for the May projections, as presented in the May WASDE report each year over 1993-2014, is listed in Table 1. Many years reflect a relatively simple approach of trend regressions, some with an adjustment for planting progress, but a few other years use a more sophisticated crop weather model (2000, 2007, 2013, and 2014). There have also been subtle changes to the trend estimation methodology when it has been used. From 1993-2006, trend estimates were based on samples that began in 1960 and generally expanded in a recursive fashion each year. Then, beginning in 2007, a switch was made to samples beginning in 1990, so corn yields from 1960-1989 were deleted from trend estimation. Then, in 2012, the yield for the most recent year, 2011, was omitted from the sample for estimating the trend regression.
While some change in procedures over time is to be expected, one comes away with the impression that WAOB corn yield forecasting procedures have been surprisingly variable. Some of these changes generated controversy and debate among market analysts at the time (e.g., farmdoc daily, May 16, 2012). The interesting empirical question is whether the changes had much impact on the WAOB yield forecasts. We present evidence on this point in Figure 1, which shows the history of May U.S. corn yield forecasts from the WAOB over 1993-2014. The dominance of using a linear trend of historical yields for making the projection for the current year is reflected in the generally steady increase in the yield projection through 2009. In fact, a linear trend regression applied to the May forecasts over 1993-2009 has an R2 of 0.98 and a slope coefficient of 2.1, which means that despite the variation in procedures over time, corn yield was simply projected to increase at a trend rate of 2.1 bushels per acre. This "steady-state" approach changed dramatically in 2010, when the May WAOB yield forecast, 165.3 bushels, jumped a whopping 8.1 bushels over the 2009 forecast. The variability that first emerged in 2010 continued through 2014.
The substantial variability in WAOB corn yield forecasts for the U.S. that has emerged since 2010 is not necessarily problematic if the methodology used to generate the forecasts is transparent and easily replicable. It would seem that a methodology limited to fitting a linear trend over average yields in a specific previous period should be transparent and easily replicated. Yet, there is the example of 2010, when the May WAOB corn yield jumped by an unprecedented 8.1 bushels but no change in methodology is indicated in the WASDE report that could explain such a large jump (see Table 1). When the methodology includes adjustment for current year planting progress and/or the application of a weather model, forecasts are less transparent and more difficult to replicate. We do note that the WAOB published a report in 2013 that presented the crop weather model used to generate forecasts in 2013 and 2014 (Westcott and Jewison, 2013). However, the WAOB has not published the data used to estimate the crop weather models nor the exact steps detailing how published forecasts are generated.
WAOB Forecast Accuracy
Since WAOB corn yield forecasts until very recently have been heavily based on trend yields, expectations for the accuracy of those projections should be influenced by the historical relationship between actual yields and trend yields. The findings in a recent farmdoc daily article (February 26, 2015) on corn trend yield projections suggests the following expectations for WAOB forecasts relative to actual yields from 1993 through 2014: 1) the average difference between projected and actual yields will be close to zero (projections are unbiased), 2) the average yield will be under-estimated more often than over-estimated, 3) there will be years with large deviations between projected and actual yields, 4) the largest deviations between projected and actual yields will be in years when actual yields are well below trend, and 5) average absolute forecast errors should be relatively small since trend explains a high percentage of variation in annual yields.
To evaluate the historic accuracy of the WAOB corn yield forecasts, the May, June, and July forecasts are compared to the "final" yield estimate released in January after harvest (we say "final" because January estimates are sometimes revised a year later or even later based on the Agricultural Census conducted every five years). The differences between the forecasts and the final estimates in percentages over 1993-2014 are presented in Figures 2 through 4. When interpreting the errors as indicated by the bars, note that a positive error implies an under-estimate on the part of WAOB and a negative error implies an over-estimate. The absolute size of the errors is also represented by the lines in each graph. As expected, the frequency of under-estimates exceeded the frequency of over estimates in all three months, forecast errors were occasionally very large, and the magnitude of the extreme over-estimates exceeded the magnitude of the extreme under-estimates. The two largest forecast errors occurred in 1993 and 2012 and are readily explained by the unusual weather conditions in these two years (floods in 1993 and drought in 2012).
The WAOB corn yield forecasts are also examined for both bias and changes in accuracy through time. The average percent errors for each forecast month for the entire sample period and for sub-samples formed from 1993-2003 and 2004-2014 are presented in Table 2. The error calculations presented in Table 2 reveal that average errors were negative (that is on average WAOB forecasts were too high) for both sub-samples for each forecast month. The average errors, however, were progressively less negative in June and July. The average errors ranged from -2.68 percent in May in the 2004-2014 period to -0.5 percent in July in the 2004-2014 period. For the entire period the average errors ranged from -2.3 percent in May to -0.92 percent in July. While average errors were negative for all months, the differences were not statistically different from zero. As a result, no statistically significant bias in estimates was found across months or sub-samples due to the relatively small size of the average errors. These results confirm the expectations that WAOB corn yield forecasts are unbiased and that average errors are relatively small.
Table 3 presents the average absolute percent errors in the corn yield forecasts for each month and sample period. As expected, the average absolute error declines modestly as the season progresses. For the entire sample period, for example, the absolute error averaged 5.98 percent for the May forecast and 5.01 percent for the July forecast. Examining the two sub-samples, the absolute percent errors were smaller in 1993-2003 for each forecast month. The increases in the average absolute forecast error in the latter sub-sample, however, are not statistically significant. Changes in absolute average errors through time are also examined by regressing the absolute percent errors against a constant and a linear time trend:
|WAOB Percent Errorm,t|= a + bTrendt + et
where Trendt is a time trend variable for crop year t that takes a value of 1 in 1993, 2 in 1994, and so on and et is a standard, normal error term. The estimated trend coefficients are negative for the May and July yield forecasts, which is counter-intuitive given the larger average errors in the latter sub-sample. The trend coefficients, however, are not statistically different from zero. The estimated trend coefficient for June is positive, but also not statistically different from zero. The trend results suggest that the major change in WAOB forecast methodology that started in 2010 neither increased nor reduced forecast errors.
As a final step in the analysis, we compare the historic accuracy of WAOB and NASS corn yield forecasts over the entire May through November annual forecasting cycle. The errors are summarized in Figure 5 using what is known as a "box-and-whisker" plot. For each month, the distance from the top of the upper whisker to the bottom of the lower whisker captures the entire range of forecasting errors over 1993-2014. The upper whisker reflects the range of forecasting errors for the largest 25 percent of under-estimates of yield and the lower whisker reflects the range of forecasting errors for the largest 25 percent of over-estimates of yield. The box captures the range of errors for the middle 50 percent of the errors. In May, for example, the WAOB forecasting errors for the U.S. average corn yield ranged from an under-estimate of 11.9 percent to an over-estimate of -32.9 percent. The middle 50 percent of the errors ranged from an under-estimate of 1.8 percent to an over-estimate of -6.7 percent. It should come as no surprise that the range of USDA corn forecasting errors gets progressively smaller across the forecasting cycle. The closer that one moves toward harvest the better is information on crop development and yield prospects. Several other interesting observations can be made based on Figure 5. First, despite the narrowing range of WAOB forecast errors from May to July, the middle 50 percent of the distribution of errors is remarkably similar. This reflects the fact that in most years over 1993-2014 only minimal adjustment was made to WAOB forecasts after May. Second, the range and middle 50 percent of NASS forecast errors during August and September were notably smaller than for WAOB forecasts. Again, this is not very surprising since NASS forecasts are generated later in the growing season and use a large-scale and sophisticated survey methodology; while by comparison, WAOB forecasts are generated earlier in the season using much simpler techniques. Third, there was some tendency for NASS forecasts in August and September to be downward biased over this sample period. This was not observed in any other month for either WAOB or NASS forecasts.
Implications
The USDA's World Agricultural Outlook Board (WAOB) provides one of the key early assessments of prospects for the U.S. average corn yield. WAOB forecasts of national corn yield are released in May, June, and July and are based on relatively simple trend analysis of historical yields sometimes modified by planting progress and/or weather conditions. Despite the simplicity, WAOB corn yield forecasting procedures have been surprisingly variable over time. The changing procedures, however, did not generate large year-to-year changes in corn yield forecasts until 2010, when the May forecast, 165.3 bushels, jumped a whopping 8.1 bushels over the 2009 forecast. The variability that first emerged in 2010 continued through 2014. Our analysis of WAOB May, June, and July yield forecasts for corn over 1993-2014 showed that the frequency of under-estimates exceeded the frequency of over-estimates in all three months, forecast errors were occasionally very large, and the magnitude of the extreme over-estimates exceeded the magnitude of the extreme under-estimates. The two largest forecast errors occurred in 1993 and 2012 and are readily explained by the unusual weather conditions in these two years (floods in 1993 and drought in 2012). Forecast error trend results suggest that the major change in WAOB forecast methodology that started in 2010 neither increased nor reduced forecast errors.
In sum, while there are no glaring problems with the accuracy of WAOB corn yield forecasts, the forecasts have been subject to criticism from time-to-time because of changing methodology, perceived inappropriate period for calculating trend, or lack of sensitivity to other potential yield indicators such as crop conditions. Some of the criticism also probably reflects a lack of understanding of the forecasting methodology. In particular, some market participants appear to be unaware of the difference between the WOAB and NASS forecasting methodologies. A substantial change was apparently made in 2010-2014, as reflected in much larger year-to-year variation in May yield forecasts. It is not unreasonable to anticipate that this changing menu of forecasting methods creates some confusion on the part of market participants. It would be very helpful if the WAOB made all data used in estimating yield models available to the public and produced a written document that outlines the exact process used to determine forecasts, including the roles of crop weather regression forecasts, subjective judgment, and any other inputs. This would go a long ways towards reducing confusion about WAOB yield forecasting methods and improving the transparency of these important forecasts to market participants.
Source:farmdocdaily