Better drag support when relation is already selected

I have a situation where I want to tidy and layout a big database diagram. In the picture above, I selected a reference. Now I want to drag it to the left. But every time I try to drag it it gets unselected, and:
- either nothing gets selected,
- or some of the neighboring references gets selected.

I would suggest improving the selection process so that when I click mouse button and hold, it does not trigger selection of references. Instead, selection should happen at mouse up event. This way I could easily drag the selected reference even if it is in a very 'crowded' place.