Update annotation without updating the LineSeries

I created a chart (with time as the X-Axis) using LineSeries and included a vertical line annotation, that scrolls through the chart according to time. Every time the position of the annotation needs to be updated, i trigger model.invalidateplot(true). As this chart is running together with other features of my application, i observed that the other features will pause when the chart is updating the annotation. Is there a way to just update the annotation and not the rest of the data (LineSeries) on the chart?