'It wasn't easy at all:' Hampton restaurant owner talks about decision to close amid scrutiny from city