A small technicality

In Capture One if you want to organize your photos inside of a catalog you have to use albums.

That depends. You could also just use folders on your hard drive - I, for one, create a folder for every day (named <yyyy>-<mm>-<dd> (<TITLE>). Or, if it is a project that spans multiple days, it gets a main folder with the project name and then the subfolders will be named per day. However, that is not a solution to the problem at hand.

Solution

Note: I tried these things with Capture One 11 + 12. I think these options were available at least since version 10, however, I have no way to test that.

When you have selected the image(s) to delete, go to the File menu (Image since v12)Thanks to @GerhardBurger for spotting this! and select Move to catalog trash or Delete from Disk. The latter option obviously deletes the file completely (Attention: Capture One does not use the OS's Recycle Bin!), while Move to catalog trash moves the file(s) into Capture One's Trash-folder (they still remain in their folders on the HDD, though). Right-click on that (or any folder or collection) and select Empty Catalog Trash. Now, a dialog will appear that lets you choose whether do delete the file(s) completely or just delete it/them from the catalog.

It should also be possible to edit the Keyboard Shortcuts (Edit - Keyboard Shortcuts... so that Del or some other key can be programmed to move the files to trash instead of just deleting the file(s) from your album.