Handling massive series

We were using MVVM to handle series containing up to a couple million points and experienced a serious bottleneck (took over a couple minutes to render and crashed on resize due to out of memory exception). The main problems were that there were too many screen points trying to be rendered, the garbage collector was being stressed too much, and the plot model was recreated upon each render. I was able to do the following:

• prevented boxing of data and screen points
• added ability to decimate
• added ability to append without re-rendering each series via a safe INotifyCollectionChanged supporting bulk addition
• allows binding directly to data if using IList<IDataPoint>
• avoids reflection and accessor functions when iterating through source data points
• avoids unnecessary recalculation of max-mins
• avoids redundant calculations in valid values
• reuses intermediate memory for rendering (to avoid stressing 3rd generational memory)
• avoids recreating model upon each render for MVVM
• I'm sure there's more
  

I did all this via inheritance, so we could live happily along side your library. However, much of this would make more sense to be added to the original source. The decimation & data appendage might make sense to leave inside a derived LineSeries, though, since they depend on monotonic x values (automatically detected of course). This done in the PCL & I only have the WPF wrappers implemented.

Would you be interested in pulling some of this in?

Yes, this is very interesting! Can you create a fork and submit each change separately? This will make it easier to review the changes.

Have you checked if the following changes could have a performance improvement

• not performing the actual clipping, but depending on the rendering capability of the output device
• clipping in a parallel loop

I have not tried deferring to the renderer for clipping or clipping in a parallel loop. I would wonder what the performance would be of creating a million point polyline even if most of those points aren't shown. If they are shown (axis is reset), then it might get tricky to do decimation depending on how much you defer to the renderer (zooming etc). I did try using a DrawingVisual instead of Polyline, but it didn't seem to make a difference. It may be because you override arrange.

I made the updates in https://hg.codeplex.com/forks/moes_leco/performance. I just ported my inheritance-based solution to one that would better fit into yours. I haven't tested the port. I do have unit tests for the Decimator, EnhancedObservableCollection, & EnhancedListCollectionView if you need, but did not add them. Really EnhancedObservableCollection & EnhancedListCollectionView may not belong in the project, but I added them so you could use to test appending without WPF misbehaving.

thanks! I started looking at your improvements! There are lots of really good ideas, but I want to review carefully before pulling them into the default branch.

I really like the change from IDataPoint to DataPoint in the root! That was something I just have to work around in my original version that plays happy with the NuGet package. It really taxes the GC to have to create and maintain millions of little objects, so I just made sure it only happened once for each series view model.

I have started pulling in your changes. Wow! You have done a great job! This made a huge improvement on performance.
I still need some time to review the last changes, I am hesitant to adding the observable pattern to the whole model...