Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for f-strings #2566

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

mrdaybird
Copy link
Contributor

@mrdaybird mrdaybird commented Feb 29, 2024

Support f-strings in lpython, because f-strings are awesome.
Previously, expression enclosed inside the f-strings was not parsed correctly. (Fixes #641 )
TODO:

  • Add new tokens to handle f-strings.
    • Divide f-strings into three parts start, middle, end to allow parsing of enclosed expression.
  • Add grammar for f-strings.
  • Add semantics for f-strings to create correct AST.
    • All the parts of f-string are merged using recursively apply add operation.
  • Implement visit_FormattedValue and visit_JoinedStr in python_ast_to_asr.cpp
    • This uses handle_intrinsic_str to cast expression to string
  • ( It works! )
  • Add tests and clean up code.
  • Update previous tests.
  • Add support(+tests) for multi line f-strings
    Maybe for another PR: (this PR is already pretty large to review i think)
  • Add escape for braces
  • Add format specifiers

@mrdaybird
Copy link
Contributor Author

@certik Can you provide me with some feedback on this PR? Are there any things that could be improved? Thanks beforehand.

@mrdaybird mrdaybird marked this pull request as ready for review March 2, 2024 08:33
@mrdaybird
Copy link
Contributor Author

mrdaybird commented Mar 2, 2024

To support f-strings I have used re2c conditions. This was the best solution I could come up with to support lexing and parsing of expression enclosed within the string.
So, this requires adding -c flag in re2c command.
@certik Also, I think the release test failing is not due to this PR. There are memory leaks while running tests on the main branch. Update: I think release test was failing due to my code. After I tried another approach without using condition mode of re2c there are no problems with the test.

@mrdaybird
Copy link
Contributor Author

I think I have finally found an efficient/clean solution to handle lexing fstrings and parsing expressions within it. I noticed re2c conditions explode the code size of tokenizer.cpp from ~3000 to ~13000 lines of code. Using multiple blocks is way more efficient and all the tests are also passing.

@mrdaybird
Copy link
Contributor Author

Now the following works:

a : i32 = 10
b : f64 = 15.5
c : str = "foobar"
print(f"Trying some calculations : {a*2+3}, {b*3.14}, some strings {c*2}")

This give the following output:

Trying some calculations : 23, 48.670000, some strings foobarfoobar

@mrdaybird
Copy link
Contributor Author

@Shaikh-Ubaid @czgdp1807 Can you please review this PR?

@Thirumalai-Shaktivel
Copy link
Collaborator

Thank you very much, @mrdaybird, for this.
I need some time to review this thoroughly. I will report back soon.

@certik
Copy link
Contributor

certik commented Apr 29, 2024

@mrdaybird my apologies that nobody reviewed it yet. @Thirumalai-Shaktivel, @Shaikh-Ubaid would one of you have time to review it please?

@Shaikh-Ubaid
Copy link
Collaborator

The changes seem considerate. I will give this a thorough look today/tomorrow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Parser] Expression tree is not properly generated in FormattedValue
4 participants