Gov. Brown: We can't let the haters win