Update: Not sure why, but today I deleted all products and started testing from scratch and it seems to be working as expected. It is necessary to have Force ID enabled for the import and I haven't tried using Reference or doing Combinations yet, but I hope it will also work.
The only "issue" I'm noticing is that I get a new image with every update. I assume it is expected as I'm feeding it an external URL, so not sure Prestashop will know that it is the same image. I'm going to try feeding it the same as the existing one, to see if Presta is smart enough to realize it is the same URL for the image and skips it.