Question about parsing the the identation language such as Python

[jbmwdzg]

@tomtau I am a new beginner on pest parser and really appreciate you and your team provide such a wonderful tool called "pest"
Currently, I am working on parsing the python with pest. Since python is a indentation sensitive language, so, I think, it should not be a good choice to add " " or "\t" to WHITESPACE of implicit whitespace, in such case, does it mean that I have to explicitly add the wsp={" " |"\t"} to each of the operator "~", it is terrible.
Do you have any suggestion on it ? really appreciate it.

[tomtau]

@jbmwdzg did you see this parser: https://github.com/thomas9911/python_parser/blob/master/src/grammar.pest ?

[jbmwdzg]

@tomtau Thanks for your quick response. After carefully understanding the sample grammar you shared, may I think that to parse the indentation-sensitive language such as python, we should not difine the WHITESPACE rule with WHITESPACE =_{" " |"\t"}, since seen from the codes within the sample, it is to take controle the newline and whitespace explicitly?
for example "function = { "def " ~ functionname ~ arguments ~ ":" ~ newline ~ block }"
In the sample codes, it only provides the case about a few scenario such as function and class, if I want to generate a whole one for python, that means, much efforts needs to be considered to control the whitespace and newline. rt?

also, for function defintion in your sample, if there is space between "functionname " and "arguments". will it fail to parse the rule "function"?

[tomtau]

Yes, I think so (to all three questions), but I haven't played with that grammar (I just know it exists)

[yshavit]

Indentation is not totally trivial. You need to handle various edge cases, like:

• dropping multiple indents at the same time:

  alpha
      bravo
          charlie
  delta
  

• dropping down to a "partial dedent":

  alpha
          bravo
      charlie  # note: less indented than bravo, but more than alpha
  

• ending the file on indentation:

  alpha
      bravo
          charlie
  delta
  

• mixed space and tabs:

   ⇥⇥ alpha
  ⇥  ⇥alpha
  

• blank lines, possibly with some spaces (but no non-space chars)
• comment-only lines
• multi-line comments, if your language supports them

You also need to decide what you do with stretches of indentation. What I found worked best when I worked on this previously (https://github.com/yshavit/antlr-denter/) is:

• newlines followed by the same indentation produce a NEWLINE token
• newlines followed by a higher indentation produce an INDENT token but no NEWLINE
• newlines followed by lower indentation produce a single NEWLINE, and then as many DEDENTs as needed
• "half-dents" produce a NEWLINE, then as many DEDENTS as needed to get to the indentation below the half-dent, and then a single INDENT

Imo, this is often done best as a pre-step, before even the lexer; but I don't know how well that would fit in with pest.

I have some work to manually turn a stream of chars into the appropriate indent/dedent/newline in rust, but unfortunately it's not really ready for sharing. But, I figured I'd at least share that research here.

[jbmwdzg]

@yshavit Thanks for sharing your idea, it is really useful.
Actually, what I plan is to develop a pre-build program to format the indentation-sensitive language such as python, which can add the boundary such as "{}", which is similar as c#, not sure whether it is feasible, will try it.

[jbmwdzg]

got it, thanks. will continue focus on the answer.