Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-32215] Correct date format for startupPosition in documentation #329

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[FLINK-32215] Correct date format for startupPosition in documentation #329

wants to merge 1 commit into from

Conversation

leijurv
Copy link

@leijurv leijurv commented May 28, 2023 •
edited
Loading

Reported on JIRA

The example string provided in the documentation does not actually parse. It causes an exception: Unable to parse date string for startup position: 2020-02-01 04:15:00.00 Z; the date should conform to the pattern yyyy-MM-dd HH:mm:ss.SSS Z

The example string in the documentation should be changed from 2020-02-01 04:15:00.00 Z to 2020-02-01 04:15:00.000 +0000

I tested this in JShell first, based on the date format defined here.

Checking the date format in JShell 
jshell> import java.time.ZonedDateTime;

jshell> import java.time.format.DateTimeFormatter;

jshell> String STARTUP_DATE_PATTERN = "yyyy-MM-dd HH:mm:ss.SSS Z";
STARTUP_DATE_PATTERN ==> "yyyy-MM-dd HH:mm:ss.SSS Z"

jshell> DateTimeFormatter STARTUP_DATE_FORMATTER = DateTimeFormatter.ofPattern(STARTUP_DATE_PATTERN);
STARTUP_DATE_FORMATTER ==> Value(YearOfEra,4,19,EXCEEDS_PAD)'-'Value(MonthOf ... 3)' 'Offset(+HHMM,'+0000')

jshell> ZonedDateTime.parse("2023-05-26 14:00:07.569 -0700", STARTUP_DATE_FORMATTER); // normal case of pacific time: works
$5 ==> 2023-05-26T14:00:07.569-07:00

jshell> ZonedDateTime.parse("2023-05-26 14:00:07.569 +0000", STARTUP_DATE_FORMATTER); // utc time represented as +0000: works
$6 ==> 2023-05-26T14:00:07.569Z

jshell> ZonedDateTime.parse("2023-05-26 14:00:07.569 -0000", STARTUP_DATE_FORMATTER); // utc time represented as -0000: works
$7 ==> 2023-05-26T14:00:07.569Z

jshell> ZonedDateTime.parse("2023-05-26 14:00:07.569 Z", STARTUP_DATE_FORMATTER); // utc time represented as Z: doesn't work!
|  Exception java.time.format.DateTimeParseException: Text '2023-05-26 14:00:07.569 Z' could not be parsed at index 24
|        at DateTimeFormatter.parseResolved0 (DateTimeFormatter.java:2106)
|        at DateTimeFormatter.parse (DateTimeFormatter.java:2008)
|        at ZonedDateTime.parse (ZonedDateTime.java:600)
|        at (#8:1)

jshell> ZonedDateTime.parse("2020-02-01 04:15:00.00 Z", STARTUP_DATE_FORMATTER); // the example string from flink documentation: doesn't work!!!!!
|  Exception java.time.format.DateTimeParseException: Text '2020-02-01 04:15:00.00 Z' could not be parsed at index 20
|        at DateTimeFormatter.parseResolved0 (DateTimeFormatter.java:2106)
|        at DateTimeFormatter.parse (DateTimeFormatter.java:2008)
|        at ZonedDateTime.parse (ZonedDateTime.java:600)
|        at (#9:1)

jshell>

Then to verify I edited statefun-flink/statefun-flink-io-bundle/src/test/resources/kafka-io-binders/routable-kafka-ingress-v1.yaml to have various values then ran mvn clean install while watching whether RoutableKafkaIngressBinderV1Test passed or failed.

Setting `type` to `date` and `date` to `invaliddateformat` (test does not pass) 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.044 s <<< FAILURE! - in org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressBinderV1Test
[ERROR] exampleUsage(org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressBinderV1Test)  Time elapsed: 0.044 s  <<< ERROR!
java.lang.RuntimeException: Error parsing an AutoRoutableKafkaIngressSpec.
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressBinderV1.parseSpec(RoutableKafkaIngressBinderV1.java:100)
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressBinderV1.bind(RoutableKafkaIngressBinderV1.java:82)
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressBinderV1Test.exampleUsage(RoutableKafkaIngressBinderV1Test.java:48)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException: Unable to parse date string for startup position: invaliddateformat; the date should conform to the pattern yyyy-MM-dd HH:mm:ss.SSS Z (through reference chain: org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressSpec$Builder["startupPosition"])
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:390)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:349)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1822)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer._deserialize(BuilderBasedDeserializer.java:489)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer._deserializeUsingPropertyBased(BuilderBasedDeserializer.java:404)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserializeFromObject(BuilderBasedDeserializer.java:319)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:219)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4569)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2798)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:3261)
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressBinderV1.parseSpec(RoutableKafkaIngressBinderV1.java:98)
	... 27 more
Caused by: java.lang.IllegalArgumentException: Unable to parse date string for startup position: invaliddateformat; the date should conform to the pattern yyyy-MM-dd HH:mm:ss.SSS Z
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressSpec.parseStartupDate(RoutableKafkaIngressSpec.java:284)
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressSpec.access$300(RoutableKafkaIngressSpec.java:58)
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressSpec$StartupPositionJsonDeserializer.deserialize(RoutableKafkaIngressSpec.java:239)
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressSpec$StartupPositionJsonDeserializer.deserialize(RoutableKafkaIngressSpec.java:217)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeSetAndReturn(MethodProperty.java:158)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer._deserialize(BuilderBasedDeserializer.java:487)
	... 36 more
Caused by: java.time.format.DateTimeParseException: Text 'invaliddateformat' could not be parsed at index 0
	at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949)
	at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851)
	at java.time.ZonedDateTime.parse(ZonedDateTime.java:597)
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressSpec.parseStartupDate(RoutableKafkaIngressSpec.java:281)
	... 41 more
Setting `type` to `date` and `date` to the example date given in the documentation `2020-02-01 04:15:00.00 Z` (test does not pass) 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.236 s <<< FAILURE! - in org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressBinderV1Test
[ERROR] exampleUsage(org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressBinderV1Test)  Time elapsed: 0.116 s  <<< ERROR!
java.lang.RuntimeException: Error parsing an AutoRoutableKafkaIngressSpec.
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressBinderV1.parseSpec(RoutableKafkaIngressBinderV1.java:100)
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressBinderV1.bind(RoutableKafkaIngressBinderV1.java:82)
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressBinderV1Test.exampleUsage(RoutableKafkaIngressBinderV1Test.java:48)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException: Unable to parse date string for startup position: 2020-02-01 04:15:00.00 Z; the date should conform to the pattern yyyy-MM-dd HH:mm:ss.SSS Z (through reference chain: org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressSpec$Builder["startupPosition"])
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:390)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:349)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1822)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer._deserialize(BuilderBasedDeserializer.java:489)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer._deserializeUsingPropertyBased(BuilderBasedDeserializer.java:404)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserializeFromObject(BuilderBasedDeserializer.java:319)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:219)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4569)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2798)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:3261)
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressBinderV1.parseSpec(RoutableKafkaIngressBinderV1.java:98)
	... 27 more
Caused by: java.lang.IllegalArgumentException: Unable to parse date string for startup position: 2020-02-01 04:15:00.00 Z; the date should conform to the pattern yyyy-MM-dd HH:mm:ss.SSS Z
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressSpec.parseStartupDate(RoutableKafkaIngressSpec.java:284)
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressSpec.access$300(RoutableKafkaIngressSpec.java:58)
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressSpec$StartupPositionJsonDeserializer.deserialize(RoutableKafkaIngressSpec.java:239)
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressSpec$StartupPositionJsonDeserializer.deserialize(RoutableKafkaIngressSpec.java:217)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeSetAndReturn(MethodProperty.java:158)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer._deserialize(BuilderBasedDeserializer.java:487)
	... 36 more
Caused by: java.time.format.DateTimeParseException: Text '2020-02-01 04:15:00.00 Z' could not be parsed at index 23
	at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949)
	at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851)
	at java.time.ZonedDateTime.parse(ZonedDateTime.java:597)
	at org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressSpec.parseStartupDate(RoutableKafkaIngressSpec.java:281)
	... 41 more
Setting `type` to `date` and `date` to the corrected format `2020-02-01 04:15:00.000 -0000` (test passes) 
[INFO] Running org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressBinderV1Test
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 s - in org.apache.flink.statefun.flink.io.kafka.binders.ingress.v1.RoutableKafkaIngressBinderV1Test

Note that just adding an extra 0 is not sufficient, it causes Unable to parse date string for startup position: 2020-02-01 04:15:00.000 Z; the date should conform to the pattern yyyy-MM-dd HH:mm:ss.SSS Z. Despite the date format ending in Z, the date must not end in Z, it must end with an actual time zone (such as +0000 for UTC / Zulu time instead of the Z shorthand).

@leijurv
Copy link
Author

leijurv commented May 28, 2023

Appears to have been added here b3a9ea1#diff-b3a7e4348711a3ca53da07698cb05263b48fed4302002ff7913f9ba51bf6287aR201

@leijurv leijurv changed the title Correct date format for startupPosition in documentation [FLINK-32215] Correct date format for startupPosition in documentation May 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@leijurv