I am not a QuickTime expert, but if it’s anything like how Windows handles video, your delay is caused by the video player determining what codec the video is using, setting up the video render pipeline (a.k.a. topology, graph, etc.) and decoding the first few frames. It will buffer those frames for you, probably in GPU (device) memory, so it’s not something you should be concerned with.
For your scenario, it must be possible to start the video player in paused mode, so that it’s ready to run in a moment’s notice. See if you can find something in the docs that allows you to do just that. If the player API doesn’t support that, you may be out of luck when it comes to the easy solutions.