Linter plugins: Add comments-related APIs

[overlookmotel]

Comment-related APIs are not yet supported in JS plugins e.g. context.sourceCode.getAllComments (as noted in docs).

The token-related APIs are going to be hard to implement, but the comment-related ones should not be too hard, as raw transfer can already deal with comments (implemented in oxc-parser).

#14488 demonstrates these APIs are required for eslint-plugin-react-hooks plugin.

[overlookmotel]

getAllComments has been added in #14589.

Next steps (stolen from @lilnasy's list on that PR):

• look into RawTransferData { program.comments, comments } duplication. (no longer relevant)
• oxlint's interface Program doesn't include the comments type. fix(linter/plugins): add comments field to TS type def for Program #14626
• Lazy deserialization of program.comments
• Remove comment.parent fix(linter/plugins): remove parent property from comments #14624
• UTF8 to UTF16 conversion on the JS side
• Test correct start/end offsets in files containing unicode chars
• getCommentsBefore
• getCommentsAfter
• getCommentsInside
• commentsExistBetween

[overlookmotel]

I'll tackle removing comment.parent. I tried just now, but it's trickier than I expected. The codegen needs some changes.

EDIT: Done in #14624.

[overlookmotel]

@lilnasy I've ticked a couple off the list. Do you have an interest in continuing work on this? (totally fine if you don't, just asking)

[lilnasy]

Yeah. I have a couple of branches I'm working on currently (see open pull requests for the ones ready to review). I'll get to comment APIs soon.

[overlookmotel]

For lazily deserializing comments:

Currently we have this in deserializer:

oxc/apps/oxlint/src-js/generated/deserialize.js

Lines 43 to 77 in 9cd7a67

	function deserializeProgram(pos) {
	let end = deserializeU32(pos + 4),
	program = parent = {
	__proto__: NodeProto,
	type: 'Program',
	body: null,
	sourceType: deserializeModuleKind(pos + 125),
	hashbang: null,
	comments: deserializeVecComment(pos + 24),
	start: 0,
	end,
	range: [0, end],
	parent: null,
	};
	program.hashbang = deserializeOptionHashbang(pos + 48);
	let body = program.body = deserializeVecDirective(pos + 72);
	body.push(...deserializeVecStatement(pos + 96));
	{
	let start;
	if (body.length > 0) {
	let first = body[0];
	start = first.start;
	if (first.type === 'ExportNamedDeclaration' || first.type === 'ExportDefaultDeclaration') {
	let { declaration } = first;
	if (declaration !== null && declaration.type === 'ClassDeclaration' && declaration.decorators.length > 0) {
	let decoratorStart = declaration.decorators[0].start;
	decoratorStart < start && (start = decoratorStart);
	}
	}
	} else start = end;
	program.start = program.range[0] = start;
	}
	parent = null;
	return program;
	}

I think we want to change:

- comments: deserializeVecComment(pos + 24), 
+ comments: null,

Then add somewhere below that in deserializeProgram:

Object.defineProperty(program, 'comments', {
  get() {
    const comments = DESER[Vec<Comment>](POS_OFFSET.comments);
    Object.defineProperty(this, 'comments', { value: comments });
    return comments;
  },
});

Those changes can be made in the raw_deser attribute on ProgramConverter here.

(with the Object.defineProperty call gated with if (COMMENTS) so it only ends up in the linter deserializer)

[overlookmotel]

Oh no, there's a further complication.

Currently deserializeWith cleans up when it's done, setting uint8 etc back to undefined, to allow the buffer to get garbage collected.

That's not going to work with lazy deserialization, because deserializeVecComment (which can get called later) needs the buffer (and sourceText) to read the comments from.

So, we'll need to:

• Alter deserializeWith so it doesn't clean up the buffer and source text.
• Add a new function reset which sets unit8 etc to undefined.
• deserialize calls reset after calling deserializeWith (so that parser does same as it did before).
• Export reset from deserialize.js.
• In linter, call reset after it completes traversing the AST (at end of lintFileImpl here).