Unripe Nectarines nibbled

Anyone any idea what could be causing this damage or how they would go about trying to find the culprit? The first year we had a good crop of nectarines but last year, most were damaged and rotted before ripening and the same looks likely to happen this year.

I've seen ants about in the polytunnel and on the tree also quite a few flies but no wasps. I do get some slug damage on other plants.