The Picture Control Utility is not compatible with low-end DSLRs (e.g. the D3?00 and the D5?00 lines), since it involves use of the function to copy Picture Control presets between the camera and the card – which was not included on these models. However, the ViewNX software which also came in the bundle supports applying Picture Controls (default or made with the Utility) to RAW files.

But what is your goal? The question as asked suggests that you are trying to import photos to your computer: any software supporting the PTP protocol will do, such as Nikon Transfer 2, most image organizers such as iPhoto, Aperture and Lightroom, or even Apple's stock Image Capture.

(Current consumer grade Nikon DSLRs can't be set to the MSC mode mentioned in the error message you got where they appear as a generic SD card reader containing image files rather than a camera that can be remote controlled and from which images can be retrieved, but this is ultimately unrelated to the inability to install the Picture Controls you made on the camera)