I hear you. It is painful: craving me time.
I have a 4 yr old, a 2 year old and a 1 year old, and I relish ibn going to work 2 days a week as it gives me more "me time" than I ever get at home.
Once a fortnight my husband drops and picks the kids up from childcare so that once a fortnight I drive off in his car (no kids seats!) and from that afternoon, after work, I am "off duty" totally. I usually go for a swim, a pamper of some description, late night shopping....then meet a gf for dinner or a movie.
It may not work into your routine with shifts etc, but worth a try?