rustc_parse: Move AST -> TokenStream conversion logic to rustc_ast

[petrochenkov]

In the past falling back to reparsing pretty-printed strings was common, so some of this logic had to live in rustc_parse, but now the reparsing fallback is only used in two corner cases so we can move this logic to rustc_ast which makes many things simpler.

It also helps to fix MacArgs::inner_tokens for MacArgs::Eq with non-literal expressions, which is done in the second commit.
r? @nnethercote

[nnethercote]

Very nice: +130 -298

@bors r+

[bors]

📌 Commit d0607449eac63ec1fcb400a77968bc399309e609 has been approved by nnethercote

[petrochenkov]

@bors r=nnethercote

[bors]

📌 Commit 09b4c7c has been approved by nnethercote

[bors]

⌛ Testing commit 09b4c7c with merge 4bb4dc4...

[bors]

☀️ Test successful - checks-actions
Approved by: nnethercote
Pushing 4bb4dc4 to master...

[rust-timer]

Finished benchmarking commit (4bb4dc4): comparison url.

Instruction count

• Primary benchmarks: 🎉 relevant improvements found
• Secondary benchmarks: 🎉 relevant improvements found

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count1	0	0	6	7	6
mean2	N/A	N/A	-0.3%	-0.5%	-0.3%
max	N/A	N/A	-0.4%	-0.6%	-0.4%

Max RSS (memory usage)

Results

• Primary benchmarks: 😿 relevant regression found
• Secondary benchmarks: mixed results

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count1	1	2	0	1	1
mean2	1.5%	1.4%	N/A	-1.7%	1.5%
max	1.5%	1.7%	N/A	-1.7%	1.5%

Cycles

Results

• Primary benchmarks: mixed results
• Secondary benchmarks: mixed results

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count1	1	1	3	6	4
mean2	2.3%	1.8%	-1.7%	-2.4%	-0.7%
max	2.3%	1.8%	-2.0%	-2.9%	2.3%

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

@rustbot label: -perf-regression

Footnotes

1. number of relevant changes ↩ ↩² ↩³

2. the arithmetic mean of the percent change ↩ ↩² ↩³

[nnethercote]

A small performance win too, nice.