Replacing 36T with 34T chain ring on Shimano FC-5800

I have the Shimano 105 Series FC-5800 front crank with the 52-36T chain ring combination on my new Canyon.

I would like to replace the 36T chain ring with a Shimano 34T chain ring from a 50-34T combination.

Is this a bad idea? Will I have trouble with the front derailleur and/or chain...