Use protected branches (only inject prod secrets on master, which can't be pushed to) and have test secrets for other branches. Now your weak spot is only anyone who can hit merge on a PR to master, which is easy to control.
On gitlab, there has been another way for some time: There is a JWT token CI_JOB_JWT available in an env var which contains the branch name and other info as one of the claims [1]. One can then use this token to obtain production secrets based on whether the branch is trusted.
Github has the same feature upcoming [2], which allows to get also directly AWS or GCP credentials restricted by branch name [3, 4].
