You can "unshare" a Collection at any time.
1) Click the Collection you would like to change sharing settings on.
2) Click the "Share" button.
3) Click the "Unshare" link next to the people you no longer want to share the Collection with.
4) Click "Update Sharing" button.
This Collection will no longer show in the Shared With Me pages of the people you unshared with.