feat: python language features #991

jingjiajie · 2024-05-19T15:05:05Z

New libs are installed, please run npm install

Summary
"Pytight" language server is integrated, which is the Microsoft's official python language server.

Thanks to the Pyright, the following features are supported:
Code completion
Hover hint
Outline for python code
and functions shown in the picture:

Due to reasons, the following features are not yet included:
Python code formatting: It's not included in the Pyright language server. It has to be a feature we separately implement in the future.
Folding range for Python code: Because we set to use the Language Server side Folding range parsing rather than using the configuration file, it only recognizes the Folding ranges we returned from language server. However, the Python folding range is naturally implemented with configuration file, so it's ignored. We have to treat it as a separate feature and manually implement it with our language server in the future.

a simple test code:

proc python;
submit;
        class Jonas:
            def name(self) -> str:
                return "Jonas"

        def salute() -> None:
            jonas = Jonas()
            print("This man is %s!" % jonas.name())
endsubmit;
run;

proc python;
    submit;
salute()

jingjiajie · 2024-05-19T15:19:16Z

@scnwwu In main branch, formatting the above test code, the code disappearing problem occurs again just like before. Would you mind to have a check again? :)

Zhirong2022 · 2024-05-20T08:13:56Z

@scnwwu In main branch, formatting the above test code, the code disappearing problem occurs again just like before. Would you mind to have a check again? :)

@jingjiajie, I have raised the issue #992 to track the changes. Thanks.

scnwwu · 2024-05-20T08:23:20Z

@jingjiajie, I failed to build your PR code on my PC.
ENOENT: no such file or directory
vscode-sas-extension\server\node_modules\pyright-internal-lsp\dist\packages\pyright-internal\typeshed-fallback
I did npm install but still no such directory

jingjiajie · 2024-05-23T03:25:41Z

@jingjiajie, I failed to build your PR code on my PC. ENOENT: no such file or directory vscode-sas-extension\server\node_modules\pyright-internal-lsp\dist\packages\pyright-internal\typeshed-fallback I did npm install but still no such directory

Fixed

scnwwu · 2024-05-29T05:21:54Z

There's a UX problem with this PR. The context menu item "Go to Definition", "Go to Declaration" etc. now always display. For most SAS code except PROC PYTHON, they have no functionality. It may be confusing for some user who rarely use Python.

scnwwu · 2024-05-29T05:23:55Z

Got unexpected "p" is not defined error with below code in the editor

proc python terminate;
submit;
#Reference to variable defined in previous PROC PYTHON call
print("x = " + x)
def my_function():
    print("Inside the proc step")
endsubmit;
run;

Got unexpected String literal is unterminated error with below code in the editor

proc python;
  interactive;
  x = '''
    endinteractive
  '''
  print('hello')
endinteractive;
run;

scnwwu · 2024-05-29T05:43:16Z

It does NOT work in browser build.

jingjiajie · 2024-05-30T01:59:47Z

There's a UX problem with this PR. The context menu item "Go to Definition", "Go to Declaration" etc. now always display. For most SAS code except PROC PYTHON, they have no functionality. It may be confusing for some user who rarely use Python.

@scnwwu It's controlled by the InitializeResult returned by onInitialize() at server side. I'm not sure if there's a way to dynamically control this. Could have a future investigation. Please let me know if you have more ideas about this design.

It does NOT work in browser build.

I found Microsoft maintained two language servers for Python, one is for desktop(Pyright https://github.com/microsoft/pyright), another is for browser(https://github.com/microsoft/vscode-python). In this PR we integrated the Pyright, so the browser version will become another work after this.
Currently I haven't had a future look at the differecies between them, but it seems that the file system is the biggest difference. Desktop version reads local python executables, libs, etc. while browser version should have totally different mechanisms. Multi-threading is another big difference. There are also many node features required by desktop version for specific features, which browser version implements differently. So that's why there have to be two version.

scnwwu · 2024-06-03T08:37:12Z

There's a UX problem with this PR. The context menu item "Go to Definition", "Go to Declaration" etc. now always display. For most SAS code except PROC PYTHON, they have no functionality. It may be confusing for some user who rarely use Python.

@scnwwu It's controlled by the InitializeResult returned by onInitialize() at server side. I'm not sure if there's a way to dynamically control this. Could have a future investigation. Please let me know if you have more ideas about this design.

Would you take a look at https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#client_registerCapability
So only register textDocument/definition when found proc python. I don't know if it will work.

jingjiajie · 2024-06-03T14:33:58Z

There's a UX problem with this PR. The context menu item "Go to Definition", "Go to Declaration" etc. now always display. For most SAS code except PROC PYTHON, they have no functionality. It may be confusing for some user who rarely use Python.

@scnwwu It's controlled by the InitializeResult returned by onInitialize() at server side. I'm not sure if there's a way to dynamically control this. Could have a future investigation. Please let me know if you have more ideas about this design.

Would you take a look at https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#client_registerCapability So only register textDocument/definition when found proc python. I don't know if it will work.

Good. It seems to work!

jingjiajie · 2024-06-03T14:46:28Z

Got unexpected "p" is not defined error with below code in the editor
proc python terminate;
submit;
#Reference to variable defined in previous PROC PYTHON call
print("x = " + x)
def my_function():
    print("Inside the proc step")
endsubmit;
run;
Got unexpected String literal is unterminated error with below code in the editor
proc python;
  interactive;
  x = '''
    endinteractive
  '''
  print('hello')
endinteractive;
run;

@scnwwu I found they are caused by Lexer.startFrom(). The CodeZoneManager reads token by calling the Lexer.startFrom() in the middle of embedded code. Without having read "proc python" in previous lines, the Lexer can't regard the token as a part of embedded code, instead it regards it as a regular proc stmt, then gives wrong result.

For example, the correct token should be 'print("x = " + x)' because it's in a proc python block, but if use Lexer.startFrom() from the start position of this line, the token returned is "print", that's wrong.

Is it possible to remove all the occurance of Lexer.startFrom() function, instead use SyntaxProvider.getSyntax()? It returns correct tokens. However it seems to be a big complexity, I'll need your confirmation.

scnwwu · 2024-06-13T09:03:40Z

Got unexpected "p" is not defined error with below code in the editor
proc python terminate;
submit;
#Reference to variable defined in previous PROC PYTHON call
print("x = " + x)
def my_function():
    print("Inside the proc step")
endsubmit;
run;
Got unexpected String literal is unterminated error with below code in the editor
proc python;
  interactive;
  x = '''
    endinteractive
  '''
  print('hello')
endinteractive;
run;
@scnwwu I found they are caused by Lexer.startFrom(). The CodeZoneManager reads token by calling the Lexer.startFrom() in the middle of embedded code. Without having read "proc python" in previous lines, the Lexer can't regard the token as a part of embedded code, instead it regards it as a regular proc stmt, then gives wrong result.

For example, the correct token should be 'print("x = " + x)' because it's in a proc python block, but if use Lexer.startFrom() from the start position of this line, the token returned is "print", that's wrong.

Is it possible to remove all the occurance of Lexer.startFrom() function, instead use SyntaxProvider.getSyntax()? It returns correct tokens. However it seems to be a big complexity, I'll need your confirmation.

@jingjiajie, please reset context in Lexer.startFrom()
Add this.context.embeddedLangState = EmbeddedLangState.NONE; under
https://github.com/sassoftware/vscode-sas-extension/blob/main/server/src/sas/Lexer.ts#L1027
Then the code zone looks correct. And the 1st snippet looks good.
I don't know what's the problem for the 2nd snippet. Would you provide more details?

server/src/sas/CodeZoneManager.ts

jingjiajie · 2024-06-18T15:40:31Z

All reported issues so far have been fixed.

scnwwu · 2024-06-19T07:05:46Z

@jingjiajie, the build is hang. Would you please address it?

server/package.json

client/src/commands/run.ts

server/src/python/PyrightLanguageProvider.ts

server/src/server.ts

server/src/python/browser/PyrightLanguageProviderBrowser.ts

Zhirong2022 · 2024-08-23T06:21:53Z

The autocomplete with item 'endsubmit' will be shown if trying to specify a value to a variable.

Zhirong2022 · 2024-08-23T06:42:19Z

It has error validation for a valid code typing.

proc python;

    interactive;
    endinteractive;
run;

tianliang657 · 2024-08-23T06:48:32Z

The autocomplete with item 'endsubmit' will be shown if trying to specify a value to a variable.

I can met the same issue with the latest extension, but it cannot be reproduced in previous extensions, so it should be a regression issue.

tianliang657 · 2024-08-23T08:23:22Z

The autocomplete cannot appear until the 'endinteractive' keyword is fully entered when entering 'endinteractive' keyword.

tianliang657 · 2024-08-26T07:20:32Z

It has error validation for a valid code typing.
proc python;

    interactive;
    endinteractive;
run;

If there is a blank line between the “proc python;” and “interactive”, the “interactive” keyword will be thrown error " 'interactive' is not defined", if the blank line does not exist, the error will not appear, this issue cannot be reproduced in previous extensions, so it should be a regression issue.

Zhirong2022 · 2024-09-03T09:01:27Z

Proper syntax highlighting for the embedded python if using the alias name of 'interactive'

jingjiajie · 2024-09-03T09:39:48Z

New reported bugs are all fixed. @Zhirong2022 @tianliang657

Zhirong2022 · 2024-09-05T08:35:27Z

There is no autocomplete if the first line is a comment. Try to type 'python' or 'print' next to 'proc'

/*  */
proc

Zhirong2022 · 2024-09-05T09:00:33Z

There is no syntax help for 'endsubmit' and 'endinteractive'.

Zhirong2022 · 2024-09-09T02:54:09Z

Suppose the proc python code written on a line intentionally as below. Delete the code and retype properly, no syntax highlighting for the SAS keyword and no outline update.

proc python;submit;print('test') endsubmit; run;

Zhirong2022 · 2024-09-09T03:41:08Z

The typed python code will be removed in the code below after format code

proc python;
    i;
    print('test')
    endinteractive;
run;

jingjiajie · 2024-09-11T09:19:54Z

There is no autocomplete if the first line is a comment. Try to type 'python' or 'print' next to 'proc'
/*  */
proc 

Fixed.

jingjiajie · 2024-09-11T09:20:08Z

There is no syntax help for 'endsubmit' and 'endinteractive'.

Fixed.

jingjiajie · 2024-09-11T09:20:21Z

Suppose the proc python code written on a line intentionally as below. Delete the code and retype properly, no syntax highlighting for the SAS keyword and no outline update.
proc python;submit;print('test') endsubmit; run;

Fixed.

jingjiajie · 2024-09-11T09:24:49Z

The typed python code will be removed in the code below after format code
proc python;
    i;
    print('test')
    endinteractive;
run;

@scnwwu I haven't read through the formatter codes. But just found a logic that is suspicious:

It seems that it discards any embedded codes. It could cause this bug.

scnwwu · 2024-09-12T05:53:51Z

The typed python code will be removed in the code below after format code
proc python;
    i;
    print('test')
    endinteractive;
run;
@scnwwu I haven't read through the formatter codes. But just found a logic that is suspicious: It seems that it discards any embedded codes. It could cause this bug.

@jingjiajie, the embedded code is handled separately. Would you please try add i beside interactive?
https://github.com/sassoftware/vscode-sas-extension/blob/main/server/src/sas/formatter/parser.ts#L131

scnwwu · 2024-09-12T05:55:59Z

server/src/sas/CodeZoneManager.ts

-          j = i - 1;
-          type = syntax[j].style;
+
+    for (let i = 0; i < syntax.length; i++) {


@jingjiajie, would you please provide a detailed explanation for this change? It looks like a fundamental logic. I'd like to verify it carefully. Thank you.

I remember this change is to fix the bug:

There is no autocomplete if the first line is a comment. Try to type 'python' or 'print' next to 'proc'

/* */ proc

Because if the starting one token of the whole document is a comment, _token() returned a null value. This change is to fix this bug.

The intention is only to fix that issue, and the remaining logics should keep the same behavior as before.

Zhirong2022 · 2024-09-12T08:03:39Z

No autocomplete if trying to type proc python or some others under comment line.

data _null_;
    set sashelp.cars;
run;

/*  */

Zhirong2022 · 2024-09-18T08:13:21Z

Error 'Request textDocument/foldingRange failed.' if trying to delete 'endsubmit' at line 6.

proc python terminate;
submit;
#Reference to variable defined in previous PROC PYTHON call
x = "Updating the value for x"
print("x = " + x)
endsubmit;

#Redefining a function called my_function
# def my_function():
#     print("Inside the proc step")
run;

proc python;
submit;
print("x = " + x)
endsubmit;
run;

Zhirong2022 · 2024-09-20T08:26:06Z

Some code will be always automatically inserted in the first python code block. Type run to load the autocomplete, select 'runtime' from the list.

proc python;
    submit;
    print('test')
    endsubmit;
run;

proc python;
    interactive;

jingjiajie · 2024-09-27T06:41:52Z

The typed python code will be removed in the code below after format code
proc python;
    i;
    print('test')
    endinteractive;
run;
@scnwwu I haven't read through the formatter codes. But just found a logic that is suspicious: It seems that it discards any embedded codes. It could cause this bug.
@jingjiajie, the embedded code is handled separately. Would you please try add i beside interactive? https://github.com/sassoftware/vscode-sas-extension/blob/main/server/src/sas/formatter/parser.ts#L131

@scnwwu Sorry for missed message, I just see your reply. I tried replace i with interactive, then it worked. I found the root cause in the code, and added support for "i" keyword in addition to interactive; Thanks.

jingjiajie · 2024-09-27T08:35:08Z

No autocomplete if trying to type proc python or some others under comment line.
data _null_;
    set sashelp.cars;
run;

/*  */

Fixed

jingjiajie · 2024-09-27T08:35:23Z

Error 'Request textDocument/foldingRange failed.' if trying to delete 'endsubmit' at line 6.

proc python terminate;
submit;
#Reference to variable defined in previous PROC PYTHON call
x = "Updating the value for x"
print("x = " + x)
endsubmit;

#Redefining a function called my_function
# def my_function():
#     print("Inside the proc step")
run;

proc python;
submit;
print("x = " + x)
endsubmit;
run;

Fixed.

jingjiajie · 2024-09-27T08:43:18Z

Some code will be always automatically inserted in the first python code block. Type run to load the autocomplete, select 'runtime' from the list.
proc python;
    submit;
    print('test')
    endsubmit;
run;

proc python;
    interactive;

@Zhirong2022
This problem is hard to fix because this behavior provided by Pyright is not configurable for us.
Pyright's rule is:
It always automatically insert import clause at the first non-empty and non-comment python code line.

In this case, the first python code line is print('test'), but actually we will insert a hidden if True: line before this line, that is, the same line of submit; clause. That's why you see it insert the import clause at the line of submit clause.

Workaround 1:
You can insert a empty line before print('test')

Workaround 2:
Another workaround is, remove indentation in the first proc python block, like this:

proc python;
submit;
print('test'); 
endsubmit;
run;

So that we'll not insert the hidden if True: line. The print('test') will be the first line.

Zhirong2022 · 2024-09-29T08:41:39Z

Suppose define a variable named 'endsubmit' or 'endinteractive', it will show SAS syntax help for the variable.
Type 'end' at the second python code block.

proc python;
submit;
endsubmit = 'hello'
endinteractive ='hi'
print('test')
endsubmit;
run;

proc python;
interactive;

endinteractive;
run;

feat: python language features

13a0fb7

jingjiajie force-pushed the python_lsp branch from 1e841ca to 13a0fb7 Compare May 19, 2024 15:10

fix: workspace init fail

b577b96

fix: doc sync & signature help

9bdc08d

jingjiajie force-pushed the python_lsp branch from 9f4c349 to 9bdc08d Compare May 23, 2024 07:53

fix: browser build problem

b8735e2

fix: dynamic menu for python

ec49b40

jingjiajie force-pushed the python_lsp branch from f7f67dc to ec49b40 Compare June 3, 2024 14:33

Jonas(Jiajie) Jing and others added 3 commits June 17, 2024 22:28

support browser

4c62d85

Merge branch 'main' into python_lsp

44bf91a

fix: code zone error for embedded code

583aa51

jingjiajie commented Jun 18, 2024

View reviewed changes

server/src/sas/CodeZoneManager.ts Show resolved Hide resolved

fix: ut hanging

24dba01

scnwwu reviewed Jun 21, 2024

View reviewed changes

jingjiajie added 2 commits June 26, 2024 21:18

fix: copyright, getFoldingBlock param struct

01280f8

Merge branch 'main' into python_lsp

61ae538

jingjiajie and others added 2 commits September 3, 2024 17:33

Merge branch 'main' into python_lsp

8038f3a

support i alias of interactive, fix codezone bugs

89cc1dd

jingjiajie force-pushed the python_lsp branch from c88174f to 89cc1dd Compare September 3, 2024 09:37

fix: endsubmit completion help, single line python

ecc5423

scnwwu reviewed Sep 12, 2024

View reviewed changes

fix: completion after comment & foldingRange crash

492b8d8

feat: python language features #991

Are you sure you want to change the base?

feat: python language features #991

Conversation

jingjiajie commented May 19, 2024

jingjiajie commented May 19, 2024

Zhirong2022 commented May 20, 2024

scnwwu commented May 20, 2024

jingjiajie commented May 23, 2024

scnwwu commented May 29, 2024

scnwwu commented May 29, 2024 • edited Loading

scnwwu commented May 29, 2024

jingjiajie commented May 30, 2024

scnwwu commented Jun 3, 2024

jingjiajie commented Jun 3, 2024

jingjiajie commented Jun 3, 2024

scnwwu commented Jun 13, 2024

jingjiajie commented Jun 18, 2024

scnwwu commented Jun 19, 2024

Zhirong2022 commented Aug 23, 2024

Zhirong2022 commented Aug 23, 2024

tianliang657 commented Aug 23, 2024

tianliang657 commented Aug 23, 2024

tianliang657 commented Aug 26, 2024

Zhirong2022 commented Sep 3, 2024

jingjiajie commented Sep 3, 2024

Zhirong2022 commented Sep 5, 2024

Zhirong2022 commented Sep 5, 2024

Zhirong2022 commented Sep 9, 2024

Zhirong2022 commented Sep 9, 2024

jingjiajie commented Sep 11, 2024

jingjiajie commented Sep 11, 2024

jingjiajie commented Sep 11, 2024

jingjiajie commented Sep 11, 2024

scnwwu commented Sep 12, 2024

scnwwu Sep 12, 2024

Choose a reason for hiding this comment

jingjiajie Sep 27, 2024

Choose a reason for hiding this comment

Zhirong2022 commented Sep 12, 2024

Zhirong2022 commented Sep 18, 2024

Zhirong2022 commented Sep 20, 2024

jingjiajie commented Sep 27, 2024

jingjiajie commented Sep 27, 2024

jingjiajie commented Sep 27, 2024

jingjiajie commented Sep 27, 2024 • edited Loading

Zhirong2022 commented Sep 29, 2024

scnwwu commented May 29, 2024 •

edited

Loading

jingjiajie commented Sep 27, 2024 •

edited

Loading