Zoom not available on axis after switching visibility

I'm not quite sure if this is a bug or I have overlooked something, if it is a bug I'll create an issue in NuGet.

Basically, I'm trying to have a feature that allows the user to switch an axis between linear and logarithmic. As suggested by Oystein in gitter, I've created the 2 Axes (one log, one lin), set one to invisible and the other to true and put my series by default to the visible one. Then, when my user clicks on something, I set the visible one to invisible vice versa and switch my series to the visible one.

It works pretty well, but neither the zoom nor the pan on the linear axis is working. I'm wondering if there is something to set at the level of the PlotModel to tell him which axis should be panned & zoomed, but I can't find any option.

Here's my code that does the switch for reference (in a setter of a property, hence the "value"):

var logAxis = this.MyPlot.Axes.Single(x => x.Key == "LogAxis");  
var linAxis = this.MyPlot.Axes.Single(x => x.Key == "LinAxis");

linAxis.IsZoomEnabled = linAxis.IsPanEnabled = linAxis.IsAxisVisible = !value;
logAxis.IsZoomEnabled = logAxis.IsPanEnabled = logAxis.IsAxisVisible = value;

this.MyPlot.Series.Apply(
	s =>  
	{
	   var series = s as XYAxisSeries;
	   if (series != null)
           {
		if (value)  { series.XAxisKey = "LogAxis";  }  
                else  {  series.XAxisKey = "LinAxis";  }  
           }  
        });