In the summer you'll want the surrounding ground to take away the heat, but in the winter that will be too cold, so you'll need to heat your home. There will be some (well quite a lot actually, as you don't have any insulation if you want ground cooling in the summer) waste heat transfering into the surrounding earth. Over time that will heat it up, so in the summer it'll be less effective at cooling.

This is actually why the London Underground is so hot in the summer. It wasn't always like this, but over time the ground surrounding the tunnels has heated up.

https://citymonitor.ai/transport/londons-tube-has-been-runni...