According to the 'Fast Diet' book (the follow on book to the eat, fast and live longer 'Horizon' documentary that was on the BBC last year) you should avoid counting calories AT ALL on the non fasting days - they found that you will generally not over eat as much as you might think, and people who ate high fat food on their non fasting days still lost weight.
Saying that, different sources say different things! So I think just do what feels right for you and try to be sensible! If you are finding it difficult to stick to less calories on the non fasting days, maybe just relax a bit and eat a little more than usual
I would recommend the book though if you get a chance to read it, it's a quick read and has some good tips!