What type of events or periodic task trigger Cpanel NGINX with Reverse Proxy to delete the cached files?

lazdeva

Member
Jun 1, 2023
8
1
3
Coral Springs
cPanel Access Level
Website Owner
According to Nginx cached files are delete based on "inactive" so expired cached files based on Expires Header or proxy_cache_valid don't get deleted until "inactive" expires and in between if the page is visited the "inactive" resets.
In the Cpanel NGINX with Reverse Proxy set up there's something that keeps deleting the cached files I would say after around 6 , 12 or up to 24 hours not honoring my "inactive_time" : "14400m", (10 days) here /etc/nginx/ea-nginx/cache.json that is correctly added by cpanel to each individual username conf file as part of proxy_cache_path.
I know, for example, that cpanel has some sort of task that overwrites username conf files and it seems there's a similar task (or tasks) run once or twice a day approximately that is deleting my cached files overwriting my nginx inactive=14400m ( I also tried inactive=10d in days instead of minutes same problem)
Could Cpanel share what are the events that trigger the deleting of all cached files everyday? Or if there's some sort of maximum imposed?
In my /etc/nginx/ea-nginx/cache.json I have "zone_size" : "200000m" which is 200gb per user, plenty, more than enough. I tried adding Nginx max_size , but is not considered by Cpanel NGINX with Reverse Proxy, but based on Nginx guidelines if no max_size is included Nginx won't impose a limit until the disk is full, so files don't get deleted until that point which is not by far the case.
There's definitely something that Cpanel is doing based on a task cron or so that deletes all files regardless of "inactive" which becomes a big problem specially when you have a big site with many cached files, sharing what are those tasks in which Cpanel will delete the entire cache will be very useful to know.
Thanks,
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
16,505
2,605
363
cPanel Access Level
Root Administrator
Hey there! I reached out to the developers about this issue since I couldn't find anything on my end, and they confirmed there is no standard scheduled clearing of the cache files that cPanel performs. As you already know, the cache entries are time-based to begin with, so Nginx will expire it automatically when it reaches that threshold. If a file changes in a way that modifies the mtime, or if an app specifically sends a "do-not-cache" header, this can alter the cached data as well.

After hearing the above from the team, I don't have a good explanation for the behavior you're seeing. If you're able to reproduce this issue, it might be best to create a ticket so we can see this happening in real-time on the system.
 

lazdeva

Member
Jun 1, 2023
8
1
3
Coral Springs
cPanel Access Level
Website Owner
OK, I have two tickets about the issue, one I closed (decided to keep monitoring myself) and another one currently open, but stuck with the analyst. I think is better when you contact the developers of Cpanel's NGINX with Reverse Proxy directly.

In regards to processes in which that Cpanel Nginx set up might be deleting cached files, I think one is, for example, when cpanel overwrites NGINX conf as part of that process it deletes all the cache, is that right? Could you confirm that the conf files overwriting process that I think is done once or twice daily won't empty the cache? I want to make sure that if I modify a conf like for example: /etc/nginx/conf.d/users/myusername/mydomain/myconf.conf cpanel as part of the modification of conf files check process won't delete the cache.

What I am looking at as a potential cause of the issue is that I noticed in the Nginx error log the following repeating like every 6 hours or so related to passenger :
[ N 2023-06-05 12:11:21.1479 46764/T8 age/Cor/CoreMain.cpp:671 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ N 2023-06-05 12:11:21.1480 46764/T1 age/Cor/CoreMain.cpp:1246 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
N 2023-06-05 12:11:21.1480 46764/T8 Ser/Server.h:902 ]: [ServerThr.1] Freed 0 spare client objects
[ N 2023-06-05 12:11:21.1480 46764/T8 Ser/Server.h:558 ]: [ServerThr.1] Shutdown finished
[ N 2023-06-05 12:11:21.1480 46764/Te Ser/Server.h:902 ]: [ServerThr.4] Freed 0 spare client objects
[ N 2023-06-05 12:11:21.1480 46764/Ti Ser/Server.h:902 ]: [ServerThr.6] Freed 0 spare client objects
[ N 2023-06-05 12:11:21.1480 46764/Te Ser/Server.h:558 ]: [ServerThr.4] Shutdown finished
...
[ N 2023-06-05 12:11:21.1899 2826/T1 age/Wat/WatchdogMain.cpp:1377 ]: Starting Passenger watchdog...
[ N 2023-06-05 12:11:21.2053 2839/T1 age/Cor/CoreMain.cpp:1340 ]: Starting Passenger core...
[ N 2023-06-05 12:11:21.2054 2839/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2023-06-05 12:11:21.3224 2839/T1 age/Cor/CoreMain.cpp:1015 ]: Passenger core online, PID 2839
2023/06/05 12:12:59 [alert] 3922#2968: ngx_http_pipelog_module: ngx_http_pipelog_command_exec(): SIGTERM detected, gracefully shutting down...
2023/06/05 12:12:59 [alert] 3046#3046: ngx_http_pipelog_module: exit_process: killpg(2968, SIGTERM) failed

Cpanel's NGINX with Reverse Proxy relies on ruby27-mod_passenger and that error that I see in the error log here: /var/log/nginx/error.logs seems that passenger shuts down NGINX. Is passenger causing all the cached files to be deleted when I see that error? It seems to coincide when I notice I get a MISS on files that were supposed to show STALE.

PER NGINX RULES NOTHING IS SUPPOSED TO BE DELETED PRIOR TO "INACTIVE" HAS EXPIRED AND I HAVE IT AS 10 DAYS, not supposed to be deleted in about 6 hours or 12 hours (approximately)
Could you have the developers to take a look at a potential bug with passenger causing this issue?

The problem with tickets is that they say "' all looks good" while they review the ticket they see the HIT, UPDATING and STALE, the problem is that I have the time and is supposed to last on STALE for 10 days , 6 hours later than the time they reviewed the ticket I get all MISS, really not going anywhere through the ticket system.

thanks,
 
Last edited:

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
16,505
2,605
363
cPanel Access Level
Root Administrator
We can definitely leave a ticket open for an extended period of time - just let us know that needs to happen and we can give it the "on hold" status so it doesn't auto close. If you already have a ticket opened, that'll be better than any information I can provider through a Forums post.
 

lazdeva

Member
Jun 1, 2023
8
1
3
Coral Springs
cPanel Access Level
Website Owner
I will really appreciate if you pass this information to the cpanel NGINX team, the ticket #95070894 seems to be abandoned
The passenger issue seems to be a bug that might be affecting many websites, specially those with Nginx Inactive greater that 5 of 6 hours, specially those sites with large number of cached pages, huge impact when you think that all is OK based on your Nginx Inactive time and instead all files getting wiped out, huge impact on Google speed factor for indexation and user interaction
thanks,
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
16,505
2,605
363
cPanel Access Level
Root Administrator
I did reach out to the devs about that, and they said Passenger itself would not be involved in the cache clearing.

I don't see that the ticket is abandoned at all, it just isn't in the high priority queue so it's not getting immediate replies.
 

lazdeva

Member
Jun 1, 2023
8
1
3
Coral Springs
cPanel Access Level
Website Owner
Any answer about this part from my prior message: "In regards to processes in which that Cpanel Nginx set up might be deleting cached files, I think one is, for example, when cpanel overwrites NGINX conf as part of that process it deletes all the cache, is that right? Could you confirm that the conf files overwriting process that I think is done once or twice daily won't empty the cache? I want to make sure that if I modify a conf like for example: /etc/nginx/conf.d/users/myusername/mydomain/myconf.conf cpanel as part of the modification of conf files check process won't delete the cache."

In reference to passenger did they get to see to error log passenger sending a signal to shut down nginx, are they sure that wont delete the cache files?

In reference to my ticket I don't see how that is not a priority remember Cpanel Nginx Reverse Proxy is now offered from cpanel comes with a manager like any other feature cpanel offers if that feature is no working as intended it should be a priority as affecting many of your users
It is a big problem to rely on that feature and your cache files get deleted, need to warn up the entire cache every 6 hours
Thanks
 

lazdeva

Member
Jun 1, 2023
8
1
3
Coral Springs
cPanel Access Level
Website Owner
OK, I will tell my hosting provider that I will cancel the Cpanel license I bought through then because that puts me at the end of the queue and I will purchase one from Cpanel directly. Thanks for the tip.
How about the questions in my previous message?
thanks for your help,
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
16,505
2,605
363
cPanel Access Level
Root Administrator
You could also reach out to your hosting provider directly as they *should* be the ones to provide the frontline support for those types of issues.

At this point, we really need to work through the ticket since this seems like a unique situation with your machine.
 

lazdeva

Member
Jun 1, 2023
8
1
3
Coral Springs
cPanel Access Level
Website Owner
I am posting this out of the obligation to warn others , after denial after denial I finally got this answer via ticket, bottom line is INACTIVE and serving STALE until INACTIVE expires, one of the most important functionalities of NGINX proxy cache is overwritten by CPANEL daily , the cached files needed to serve the fast STALE version while updating , unless is less that 1 day becomes useless, if you have 1 day or longer in your INACTIVE in this file /etc/nginx/ea-nginx/cache.json as per here: NGINX with Reverse Proxy | cPanel & WHM Documentation won't work. Part of the reply I got:

"By default, cPanel will remove the cache after reloads. Reloads will happen every day as part of log rotations. A reload could also occur during package updates for Nginx."
" But in addition to this, the cache on storage will be missing every day as part of cPanellogd functions or reloads sent to Nginx. To ensure the cache stays valid longer, the TTL for cache valid will need to be increased. But developers intended for the cache to not remain in the storage longer than one day. "


As you can see the 1 day thing from the Cpanel Nginx Proxy developers makes this CPanel NGINX proxy cache useless if you have more that 1 day in INACTIVE, I learned the hard way until getting provided proper disclosure now.
One of the most important features of NGINX PROXY CACHE which is the option to have a long INACTIVE (10 days for me) for the cached files to be served as STALE (fast) while updating is completely overwritten by cpanel, if you think your site is serving STALE cached files based on INACTIVE longer that one day , those cached were deleted by CPANEL in 1 day, website visitors will get a slow MISS

Hope this helps others
 
  • Like
Reactions: cPRex