Having been to Portugal numerous times in April I find the daytime temperature perfect for sunbathing and swimming without being stifling. Whilst it can be a bit cool when the sun goes down, by the time we reach May the evening temperature usually maintains at around 20C, which is very pleasant.
Especially if you have small children, being on the Algarve in August can be a gamble, last time we went then it was 46C and consequently we spent most of the afternoons inside, because its just too hot and too much for little people.
So take the opportunity to visit now, the month of May really is a great time for those with young families and babies.
