How bad is it to only really eat on the weekends, and expensive restaurants at that?

I graze during the week day due to stress, mostly. On the weekends, I eat out with friends to calorie laden restaurants. How bad is that?