Cache Drupal slow after x time

Hello everybody.

I have a “problem” with cache of Drupal.

When I purge the site is slower, it is normal.

It is the time that the cache is rebuilt. When the cache is rebuilt and the site is often visited all is perfect!

1 sec to load a page about 1 MB, I’m very happy about that.

But when the site is not used by the user for few hours it seems that the cache is automatically rebuild …

2.5 sec by page.

In my configuration performance in Drupal I set cache lifetime to none.

Anymore configuration in Php that I miss? Or anything else ?
i precise that no additional module is installed for the cache.

Thank you for your help

Drupal version: