Should I let this bother me?

My husband and I have an 8 month old. I work a 9-5 job while he stays at home and takes care of our child during the day. He works part-time on the weekends and collects unemployment. I'd rather be at home taking care of the baby; but someone has to bring in the income, right? I came home the other day and he told me that I would have to cook something for myself because he wasn't hungry. I was like ok, that's fine, whatever. No big deal. So I asked him if he went and got food earlier, because he's typically starving by the time I get home. He said, "No, I just had a sandwich. I'm not hungry." Well last night I found out that he and his sister in law (his brother's wife) went out and had lunch together. I asked him why he didn't just tell me that and he said, "I don't know. I just didn't." I'm still trying to figure out why he lied to me about it. Should I start getting worried? His sister in law likes to come over a lot during the day to see our son. I told him that if I pay the mortgage, I have a right to know who is coming into my home. Should I let this bother me? I feel like if he is willing to lie about things that are so trivial, who says he won't like about more significant things?