There was a saw mill there once. Also people would have to move logs to clear the property to build on. As well as that they have already found theses parts of logging poles on the island before.
J.Steele believes the island was used to make naval stores in her book This would of had heaps of logs/trees being moved about.
https://www.oakislandcompendium.ca/sawmill.html
On a related point the map shows the sawmill seems to be where they found those rocks with the holes in them.
This would probably explain the rocks with the holes and spikes in them as well.