I suspect that this is not possible, but I was wondering if there is any way to only have one line out (RCA vs XLR) active at once and switch between them?
With the current firmware, no.
The reason is that if I wanted to try out a different headphone amplifier for a while via RCA and use this purely as a DAC, I'd like to still be able to switch to my studio monitors that are currently plugged in via XLR.
Unless you would want to use both the DAC's built-in headphone amplifiers, you could repurpose one of them as a line out (you might also be able to use both in case you were mad) using a stereo-jack (TRS) to 2xRCA adapter cable.
Going into 'Setup -> Device Mode -> Mute Line' and selecting 'Toggle plugged' cycles through the connected outputs (in the front) and the Line Out by long pushes of the volume knob.
If you want to send signal to both the headphone amplifiers, select 'Toggle Ph/Line' instead. This will switch between the front and the Line Out. This way the volume control for both front outputs is linked.
Keep in mind, however, that the level of IEM output at 0 dBFS is only -3 dBu (0.55 V RMS) while typical levels for RCA line outs are around 2 V RMS (8 dBu). Thus, the connected headphone amplifier needs to have enough gain to make up an 11 dB level difference.
If you use the Phones output as a line out, you might have another problem as the output level at 0 dBFS with Hi-Power enabled is 22 dBu (10 V). This is 14 dB (5 times) higher than typical RCA line outs and you may damage the amplifiers input stage or you could only use the lowest range of the volume control if the gain of the amp is too high.
The best solution would be to use Phones output at Lo-Power where the output level at 0 dBFS is 7 dBu (1.73 V RMS) which is in line with typical RCA line outs and thus likely the best match to the gain of an external amp.
As far as I can tell, this is not possible and I would have to use a downstream XLR switcher, but then I lose the awesome ability of the ADI-2 DAC to remember all my different outputs' settings when I switch.
Thanks for any advice
Those settings can be store in different setups which can activated using the press of a button using the DAC's remap keys function.