Dyeing Balayage Blonde Back to Brown?

I have been growing my hair out after dyeing it blonde from brown and recently (two days ago) I got q balayage done to blend my roots. I would really like to dye my hair back to brown now. How long should I wait? Can I do so later this week or will that be too much damage?