You can "unshare" a Moment at any time.
1) Click the Moment you would like to change sharing settings on.
2) Click the "Share" button.
3) Click the "Unshare" link next to the people you no longer want to share the Moment with.
4) Click "Update Sharing" button.
This Moment will no longer show in the Shared With Me pages of the people you unshared with.