Replacing interior wall panels

In the master bathroom of my Westgate Eichler, the previous owners cut into the paneling and installed a 'built-in' (i.e. recessed) oak towel rack and also a toilet roll/magazine holder. The paneling is painted so I'm not looking to replace them with mahogany, but I'm wondering how easy it is to replace a piece of paneling and if people have done this with a standard non-grooved paneling? I'd like to do this and install more modern fixtures.

Thanks