How can I prevent the foreign key column from being deleted when I delete a foreign key relationship?

You can't. Deleting a foreign key automatically deletes the corresponding foreign key columns. But! There's a workaround: you may edit the reference so it contains no columns, and then delete it.

First, select the reference.

Then, remove the column the reference contains: