While two questions, I think they have a common thread that unites them: the need to let Lightroom do everything, not trying to "help" by manual changes and moves.

As to removing the photo, the best plan with Lightroom is to do so in Lightroom. So if you wish to remove a photo entirely, do it without Lightroom and when prompted use the "Delete from disk". Doing anything separate from lightroom will tend to get the catalog and your PC folders out of sync.

As a related note, most people feel you do not need to keep lightroom exports in the catalog at all, e.g. if you export JPG's for some purpose, just export and delete when done. You can always recreate the export with a couple clicks. Trying to then import them back into the catalog (as some people do) is rather pointless. (Distinguish this from derivative edits done elsewhere, e.g. if you export to edit in another photo editor, then you may want to import it back, since you cannot reproduce it directly from Lightroom.).

As to the history of changes, the development changes can be written to a "sidecar" file, otherwise known as an XMP file, by using the Write Metadata option (you can also set, in the preferences, lightroom to do this all the time). It creates an additional file in the same folder and same name (except for XMP file type) as the original, which contains all the development settings. This can be imported into a different copy of lightroom, as well as Photoshop, and some few 3rd party programs (though typically there for metadata not develop settings). If you are going to a different Lightroom instance, you may want to export as a catalog, as this can be simpler and more automated, but both work.

It sounds though as you are trying to remove raw files to save space - you can do that "inside" of lightroom as well, in the sense you can use Lightroom to move (by drag and drop) them only an external drive. Lightroom is able to deal with external drives that are not on line at a given time, and are mounted and dismounted as needed. Moving to these drives inside lightroom is the best way to ensure the catalog and folders stay in sync - do not move things around manually outside of lightroom. You will likely need to create an empty top level folder first (Lightroom folder panel), then can drag the folders needed and drop them in it. By storing the files on offline drives, but leaving them in the catalog, you have no need to export settings or history, since that is still in the catalog (and quite small). Just mount the drive, and edit away picking up where you left off, with no loss of info.

Removing from the catalog, even with the XMP file attached, loses some information such as collection membership.

Be sure you back up the external drives, of course; some people tend to forget them while backing up their internal drives. External drives do fail, probably more frequently than internal.