I found a great article from William Lam about how to build an Photon OS virtual appliance which includes OVF settings for configuration. During my initial tests I faced some issues like the virtual machine network did not connect properly and others. First I had changed the existing scripts to work with the vmware-iso builder which was in the end working but the performance was pretty bad. So I decided to convert the existing template from vmware-iso to vsphere-iso.
The base code is really good an most of it can be re-used. My changes can be found in my fork of his repo.
And here a video of a build run.