If you are not too concerned with the global structure of the genome, helped (in many cases) by the scaffolding, then just use the contigs and save yourself any hassle.

However, I generally think the concerns in using the scaffolded assembly are whether the scaffolds are correct (probably more of an issue with more repeat content, ploidy complexity etc) and whether the downstream annotation would have an issue with the N's

I second the advice you got already - use contigs.fasta. Also, it would be worth checking whether there is any difference between the two files. Sometimes they are identical, so there is no choice to be made.

In most cases there are minute differences between the two files. It is not like anything major will be gained or things will be screwed up by using one file vs the other.