The main disadvantage I have seen with silent mode is that the shots/frames per second rate in continuous mode is reduced. E.g. the Canon EOS 7D Mark II can only take 4 shots / second in silent mode, where in non-silent high speed it can take 10 shots / second. Also, there is an even longer delay from the time you push the shutter button to when the photo is taken (also known as "shutter lag"), but one could argue that it is not noticeable.

If you are using your camera for sports, it might be better to not use silent mode if you want to use continuous for burst shooting.

For portraits, you probably will not even know the difference between silent / non-silent. Your subject should be siting still enough so that the photographs between the 2 modes would hardly be noticeably different.

Silent is best used in places where you do not want to have the shutter sound being predominant, like in some churches or events where you need to be quiet.

I also acknowledge that Canon's cameras are not exactly the same as Nikon's, but the concepts of silent mode are essentially the same.