Normally we plunge through the material and then start the circular motion. This means we are cutting the full material thickness at the end of the cut. Any movement of the stock will cause a slight divot or mark.
You will get a smoother cut by starting the circular motion as you enter the wood and plunge through on a gradual angle.