The S&P 500 Is the Most Expensive It's Been in Decades. History Has Good and Bad News for Investors

The S&P 500 (SNPINDEX: ^GSPC) index has been on a phenomenal run in recent years, with the broader benchmark more than doubling since the start of 2023.

Investors have grown their wealth significantly in this time, but all success comes with a price. In this case, the market now looks quite expensive. In fact, the S&P 500 trades at its most expensive valuation in decades.

Many investors are now bracing for a correction or pullback. History has both good and bad news.

Image source: Getty Images.

Bad news: a big pullback is likely coming

Well, the obvious bad news for investors is that, based on what the market has been doing and signaling this year, history suggests that the market could be headed for a significant correction, a bear market, or even worse.

Now, a correction, in which the market falls 10%, is actually not all that uncommon and happens more than you think. But if we look back at history, there's also a chance the market could undergo an even bigger reset.

Looking at the Shiller CAPE ratio, which divides the S&P 500 by its average 10-year, inflation-adjusted earnings, the market is as expensive as it has been since before the Dot-Com Bubble.

The Shiller CAPE ratio uses a 10-year average of inflation-adjusted earnings to smooth out volatility the market encounters over the course of an entire business cycle.

Data by YCharts.

The whole Dot-Com era proved to be one of volatility. Between 1995 and 2000, the tech-heavy Nasdaq Composite (NASDAQINDEX: ^IXIC) surged about 500%. Once the Dot-Com Bubble burst, the Nasdaq lost 77% of its value by October 2002.

It would recover some, only to take another big hit during the Great Recession.

Given the similarities between the internet-fueled Dot-Com Bubble and the current artificial intelligence supercycle, investors should brace for a significant pullback.

Now, the saying goes that while history often rhymes, it rarely repeats. The fact that so many investors are bracing for a pullback makes me believe it may not happen in the near term or may occur due to some unforeseen circumstance.

It may also happen much faster than in the past, as we've seen with big sell-offs in recent years. They happen quickly, and the market rebounds quickly, so perhaps that's the new reality.

Investors concerned about a big sell-off can hedge by diversifying some of their holdings into sectors less affected by AI or into an equal-weight S&P 500 fund, which is not market weighted.

Good news: investors should be OK in the long term

The good news is that history shows the market tends to rise over the long term, and there's no reason to believe it won't continue to do so.

Even though the market and internet stocks got crushed in the early 2000s, look at where we are today. The internet did indeed change the world, and many companies that recognized this early on ended up doing extraordinarily well.

However, some of the first companies that tried to take advantage of the internet didn't do so well, and it took longer than many expected for the internet to work.

A similar tale could play out for AI. Perhaps it won't be Anthropic or OpenAI that ends up being the winning artificial intelligence models. Perhaps even some hyperscalers will be eclipsed by other companies.

The overall lesson for investors is to assess where you are in your investing journey. If you are older, plan to retire in a few years, and want to focus on capital preservation, the S&P 500 may not be the best place to invest.

If you are younger and have a 10- to 20-year runway ahead, then you can stay invested in the market and be more aggressive. However, if you own individual stocks, it's always good practice to look at valuations and assess whether the company can continue to grow earnings steadily and consistently over the next decade.

Bram Berkowitz has no position in any of the stocks mentioned. The Motley Fool has no position in any of the stocks mentioned. The Motley Fool has a disclosure policy.