The second homes crisis in Wales is likely to get worse because of Brexit and Covid-19, a new report has warned.