Hi, flo62134. Indeed there are multiple explanations for a miss with varnish cache as you said.
Varnish cache works with objects and it's based on urls, so that if you call the same object from multiple urls you will have duplicated objects in your cache.
In my case, I had to normalize all my urls prior to achieving a high hitrate in varnish. A common example is to redirect all www.yoursite.com traffic to yoursite.com or the other way around. If your varnish sits behind your web server, you'll have to make the redirections on varnish, otherwise it may always have a miss on the first request.
In conclusion: Check the first request in terms of objects and urls. If you find what's different between the requested and the cached object you'll find what the problem is.