Why does this matter? Because AFAICT the current best practice for implementing languages is to 1) prototype the language with a tool, then 2) throw that out and write the parser by hand.
If you skip step 1, then it will be unclear what language you've defined (and it's not always obvious how to break up the procedures). If you skip step 2 and auto-generate your code instead, you typically don't have enough expressive power, and you have to hack around it (communication with globals, etc.)
Tools and theory are focused excessively on CFGs (i.e. in this thread parsing with derivatives). Whereas real language always have non-context-free parts. Granted, Python is probably one of the most friendly and well-designed, and writing the lexer by hand and generating the parser works well there. I don't think that, in the case of Python, writing the parser by hand would have been a good idea.
Not the author, just wrote a Python parser recently. The problem with most grammar specifications is that they are not clear about the tokenization stage and assume that you already have a valid token stream to work with, which is of course not the case usually. The Python grammar specification for example does not even mention anything about how one should implement the tokenizer (to my knowledge).
That said, most programming languages can be formulated in an almost context-free way.
For real-world parsers, I think speed is just valued much higher than elegance, which often makes sense, especially for scripting languages where the parsing speed is directly proportional to program startup time (leaving caching aside).
Agreed, although I would not say "most" can be formulated as CFGs. There is the lexer issue you mention, and that is mentioned in that article, but there are also tons of other issues in real languages.
Languages that are close to context free: Python, Java, JavaScript, R, Go, awk. (and I think Standard ML and OCaml)
Languages that are not even close: C, C++, Perl, Unix shell, make.
Languages I'm not sure about but which probably fall in the "no" category: PHP, Ruby, C#.
So I think that is enough to warrant some research into parsing tools for non-CFGs. (PEGs are interesting and I explored them for some of these applications)
http://trevorjim.com/python-is-not-context-free/
(Are you the author? :) )
Why does this matter? Because AFAICT the current best practice for implementing languages is to 1) prototype the language with a tool, then 2) throw that out and write the parser by hand.
If you skip step 1, then it will be unclear what language you've defined (and it's not always obvious how to break up the procedures). If you skip step 2 and auto-generate your code instead, you typically don't have enough expressive power, and you have to hack around it (communication with globals, etc.)
Tools and theory are focused excessively on CFGs (i.e. in this thread parsing with derivatives). Whereas real language always have non-context-free parts. Granted, Python is probably one of the most friendly and well-designed, and writing the lexer by hand and generating the parser works well there. I don't think that, in the case of Python, writing the parser by hand would have been a good idea.