3
$\begingroup$

I want to preform a machine learning task (e.g. supervised classification, clustering) on a corpus of programming language source code (lets say Python), and I'm looking for tools for parsing and constructing structures out of a document of Python (for example) code, similar in concept to NLP tokenization and higher level language processing, adjusted for programming languages.

While I'm able to find a lot of general NLP related material, it seems to me a better starting point would be previous research specifically focusing on programming languages.

I'm looking for tools, resources, academia articles and keywords to search for, basically any help is appreciated!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

NLP stands for Natural Language Processing. Programming language source code are synthetic (or unnatural) languages. Thus, NLP tools are not useful for processing programming language source code.

Understanding programming language source code is done by the compiler or interpreter. Compilers and interpreters perform many functions, including lexical analysis, parsing, and semantic analysis.

For Python language, language analysis is relatively straightforward since the main Python implementation, named CPython, compiles Python programs into intermediate bytecode, which is executed by the virtual machine. The dis module supports the analysis of CPython bytecode by disassembling it.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I'm aware of the dis module and the concept of compiler, interpreter and byte code translation (heck, I wrote and reverse engineered byte code translators and interpreters). This doesn't quite address the goal of statistically analyzing languages (python being an example) $\endgroup$
    – NirIzr
    Commented Aug 21, 2017 at 5:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.