There’s no one answer here. It’s going to take a lot of trial and error and experimenting. All of the issues you mention are going to have to be addressed individually as well. There is never going to be a single tool to do this for you.
As far as tracking state over time, standing up a proper, modern monitoring stack will help tremendously. If you send logs to loki, collect metrics with Prometheus or OpenTelemetry, and graph them both with grafana, you should have really great insights to whatever is happening… It’s never going to be finished though. It’s always a work in progress.