Merge #59

59: [DOC] Specification tests PR3 r=saem a=haxscramper



Co-authored-by: haxscramper <[email protected]>

.gitignore

@@ -110,3 +110,7 @@ htmldocs
 nimdoc.out.css
 # except here:
 !/nimdoc/testproject/expected/*
+
+# For anyone who wants to make a workspace out of nimskull repo
+*.code-workspace
+.vscode

tests/lang/readme.md

@@ -0,0 +1,114 @@
+# Language Specification Tests
+
+Contained within are a series of tests which ensure the language behaves exactly as documented. You can consider it to be a form of documentation in itself.
+
+## Organization
+
+Specification is structured and written using bottom-up approach - first we
+specify the most fundamental language elements, such as string and integer
+literals, variables. Then we add scopes, control flow-altering constructs
+and so on. When new language construct is added, it first is described
+separately, and then, if needed, revised again to account for preexisting
+language features. Separate language constructs should be placed in
+different sections or files, but "revisions" can be exemplified in the same
+test file.
+
+You might think of specification as an aid to help you to write an own
+implementation of nim, or support a new backend. First you implement the
+most basic language features, then you progressively iterate them to
+correctly match their behavior with everything that has been added prior to
+that.
+
+Example of how decision about file structuring is made - we need document
+implementation of the iterators - they can return values, overload on
+arguments, their behavior can be altered using keywords such as `break`,
+`continue` and `yield`. This is already enough for us to decide that this
+is not a "basic" feature, and formulate the prerequisites that language
+implementation must have before implementing iterators.
+
+1. Language must support overloading, which must've been tested in the
+   "procedures" section.
+2. `break` and `continue` must be at least recognized by the parser - their
+   behavior might be different inside of an iterator body, so their
+   behavior on `while`/`block` and so on is only important for the sake of
+   consistency (you can *expect* `break` to work with iterators the same
+   way it does with `while`)
+
+When it comes to writing a test for the iterator itself we first start with
+the most basic iterator that does not accept any arguments, and can only
+`yield` values. We check if order of execution for statements is correct,
+and then `yield` does really return the same value. Iterator must also
+preserve it's state between execution transfers - we test it too.
+
+Then we got to check how iterator implements `break` and `continue`, then
+overloading and so on.
+
+Different language constructs should be specified and tested in the same
+order - first you imagine adding new feature to the language, specify it's
+core properties (`yield`, control transfer, state preservation), then
+expand the definition by providing a specification for interactions with
+other language constructs (`break`, `continue`, `defer`).
+
+The only part of the specification that does not follow these rules is
+"atoms" section, that describes fundamental capabilities of the language
+such as *expressions*, *statements*, *type*, as well as a constructs used
+in the tests themselves.
+
+
+## Contributing tests
+
+To add new tests please use following template
+
+```nim
+discard """
+description: '''
+Description of the test itself. Brief list of checks
+'''
+"""
+
+## Language feature documentation.
+
+block example1:
+  # ...
+
+
+
+```
+
+Please note that it is strongly encouraged to comment the tests as much as possible; the main objective is not simply *"list all possible combinations of all features"*; the objective, is instead, to provide a ***literate executable specification*** of the language.
+
+## Error tests & specifications
+
+It is also necessary to document compilation and runtime errors within, preferably with examples on how to fix the particular issue.
+
+For error reporting you can use following template:
+
+
+```nim
+discard """
+description: '''
+Test description
+'''
+
+errormsg: '''
+Actual error message
+'''
+"""
+
+## Detailed! explanation for the error cause and possible solutions to it.
+
+block working_example:
+
+# One or more examples of working code
+
+block failing_example:
+
+# Example of the failing code
+
+```
+
+
+## File naming
+
+- `t01_feature_name.nim` - start file name with number to explicitly order features.
+- `t01_feature_name_fail.nim` - show example of the runtime failure

tests/lang/s01_basics/README.md

@@ -8,10 +8,9 @@ would not think to test. Mostly this captures minor trivia around:
 - if, while and case statements
 - code block and scopes
 - primitive expressions
-- Pure-nim user-defined types, without going into details about interfacing
-  with C and C++ objects, pragmas and any other complications.
+- Primitive data types such as strings, characters, booleans and integers.
 
 Tests in this section are mostly used to document how most basic interactions
-are done - stuff you would find in almost any regular imperative programming 
+are done - stuff you would find in almost any regular imperative programming
 language. Things that are more nim-specific should be placed in `s02_core` or
 other sections.

tests/lang/s01_basics/s00_atoms/readme.md

@@ -2,12 +2,12 @@
 
 This directory contains specifications for basic language components:
 
-- What is an expression
-- What is a statement
-- What is "compile time"
-
-For a lot of these, it is difficult to demonstrate without using
-concepts from subsequent specification tests; but, it was decided to put them
-as a first entry in the specification. When more "advanced" entries are used,
+- expression
+- statement
+- "compile time"
+- "type"
+
+For a lot of these, it is difficult to demonstrate without using concepts
+from subsequent specification tests; but, it was decided to put them as a
+first entry in the specification. When more "advanced" entries are used,
 these should be referenced.
-

tests/lang/s01_basics/s00_atoms/t04_type_inference_fail_colon_expr.nim

@@ -0,0 +1,26 @@
+discard """
+description: '''
+Type inference does not work if rhs is a complex expression
+
+Also potentially related - https://github.com/nim-lang/Nim/issues/7321 although
+```nim
+var s: seq[Shape] = @[x]
+```
+
+is actually a non-trivial expressions, since it does `var s: seq[Shape] = @([x])` -
+which calls magic/generic `@`.
+
+'''
+knownIssue: "https://github.com/nim-lang/RFCs/issues/149"
+knownIssue: "https://github.com/nim-lang/Nim/issues/11777"
+"""
+
+block infer_set_type:
+    # where this one does work
+    var y: set[char] = {}
+
+    # this one doesn't work (crashes the compiler)
+    var x: set[char] = (discard 12; {})
+
+    # this crashes compiler as well
+    var q: seq[int] = (discard 1; @[])

tests/lang/s01_basics/s05_data_types/t1_integer.nim → tests/lang/s01_basics/s05_primitive_types/t01_integer.nim

@@ -15,7 +15,7 @@ target: "c cpp js"
 # - narrowing conversions for `int` literals (compile time) are compile time
 #   checked, and if they fit are done implicitly
 
-block:
+block type_conversion_to_8:
   let
     a: int8 = 1
     min: int8 = -128
@@ -25,7 +25,7 @@ block:
   doAssert typeOf(max) is int8, "auto-narrow: literal `int`->`int8` - max"
   # xxx: cover the over under flows in separate tests
 
-block:
+block type_conversion_to_unsigned_8:
   let
     a: uint8 = 1
     min: uint8 = 0
@@ -35,7 +35,7 @@ block:
   doAssert typeOf(max) is uint8, "auto-narrow: literal `int`->`uint8` - max"
   # xxx: cover the over under flows in separate tests
 
-block:
+block type_conversion_to_16:
   let
     a: int16 = 1
     min: int16 = -32_768
@@ -45,7 +45,7 @@ block:
   doAssert typeOf(max) is int16, "auto-narrow: literal `int`->`int16` - max"
   # xxx: cover the over under flows in separate tests
 
-block:
+block type_conversion_to_unsigned_16:
   let
     a: uint16 = 1
     min: uint16 = 0
@@ -55,7 +55,7 @@ block:
   doAssert typeOf(max) is uint16, "auto-narrow: literal `int`->`uint16` - max"
   # xxx: cover the over under flows in separate tests
 
-block:
+block type_conversion_to_32:
   let
     a: int32 = 1
     min: int32 = -2_147_483_648
@@ -68,7 +68,7 @@ block:
 # at this point, it starts depending upon the platform whether it's a narrowing
 # or widening of the type
 
-block:
+block type_conversion_to_unsigned_32:
   let
     a: uint32 = 1
     min: uint32 = 0
@@ -78,7 +78,7 @@ block:
   doAssert typeOf(max) is uint32, "auto-conv: literal `int`->`uint32` - max"
   # xxx: cover the over under flows in separate tests
 
-block:
+block type_conversion_to_64:
   let
     a: int64 = 1
     min: int64 = -9_223_372_036_854_775_808
@@ -91,12 +91,40 @@ block:
 # automatic/implied widening and contrast with narrowing
 # xxx: need to add more coverage
 
-block:
+block type_conversion_to_unsigned_64:
   let
     a = 6'i8
     b: int = 10
     narrow = a + 10
     widened = a + b
   doAssert typeof(narrow) is int8, "integer operations with literals narrow"
   doAssert typeof(widened) is int, "runtime integer operations widened types"
-  doAssert narrow == widened,      "the values are comparable and equal"
+  doAssert narrow == widened,      "the values are comparable and equal"
+
+block negative_literals:
+  doAssert -1 == -(1)
+
+  doAssert -1'i8 == -(1'i8)
+  doAssert -1'i8 == -(1'i8)
+  doAssert -1'i16 == -(1'i16)
+  doAssert -1'i16 == -(1'i16)
+  doAssert -1'i32 == -(1'i32)
+  doAssert -1'i32 == -(1'i32)
+  doAssert -1'i64 == -(1'i64)
+  doAssert -1'i64 == -(1'i64)
+
+
+block literal_prefixes:
+  ## Integer literals may be given in decimal (no prefix), binary (prefix `0b`), octal
+  ## (prefix `0o`), and hexadecimal (prefix `0x`) notation.
+  block binary_literals:
+    doAssert 0b10 == 2
+    doAssert 0b00 == 0
+
+  block octal_literals:
+    doAssert 0o1 == 1
+    doAssert 0o10 == 8
+
+  block hexadecimal_literals:
+    doAssert 0x1 == 1
+    doAssert 0xA == 10

tests/lang/s01_basics/s05_primitive_types/t04_char.nim

@@ -0,0 +1,6 @@
+discard """
+disabled: true
+description: '''
+Placeholder file just in case, covered elsehwere
+'''
+"""

tests/lang/s01_basics/s07_operators/t1_operators.nim → tests/lang/s01_basics/s07_operators/t01_operators.nim

@@ -43,22 +43,15 @@ targets: "c cpp js"
 # 7  - `&`                      concatenation
 # 6  - `..`                     slice
 # 5  - `== <= < >= > != in      comparison
-#       notin is isnot not of   
+#       notin is isnot not of
 #       as from`
 # 4  - `and`                    logical and/multiply
 # 3  - `or xor`                 logical or/sum
 # 2  - N/A                      overloading related
 # 1  - `=`                      assignment
 # 0  - N/A                      arrow-like
 
-# mostly covering precedence here as definition and overloading (AKA later) is
-# where much of this becomes relevant
-
-# xxx: expand for more exhaustive coverage, while avoiding adding too many
-#      features that have not been tested in previous secions.
-
 doAssert false == false, "`==` is composed of the `=` token in a row"
-
 doAssert $1 == "1", "to string operation"
 
 doAssert 2 + 2 * 10 + -2 / 2 == 21,