Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errors when s3 caching #877

Open
btwiningfc opened this issue Aug 8, 2022 · 1 comment
Open

Errors when s3 caching #877

btwiningfc opened this issue Aug 8, 2022 · 1 comment

Comments

@btwiningfc
Copy link

I've been running a simple caching process using version 10.4 and PostgreSQL 11.X with PostGIS 2.5 for a couple of years with no issues. I used the postgis provider

I recently upgraded to Tegola 15.2, postgreSQL 14 and PostGIS 3.1.

Seed command:
/opt/tegola cache seed --bounds $BOUNDS --config $CONFIG_URL --min-zoom 14 --max-zoom 14

Config toml:

[webserver]
port = ":9090"
uri_prefix = "/tile-server"

[webserver.headers]
    Access-Control-Allow-Origin = "*"
    Cache-Control = "no-cache, no-store, must-revalidate"

[cache]                     # configure a tile cache
type = "s3"                 # will cache to the AWS S3 bucket
bucket="bucket"
aws_access_key_id="KEY_ID"
aws_secret_access_key="SECRET_KEY"


# register data providers
[[providers]]
name = "test_postgis"  
type = "postgis"       
host = "host"          
port = 5432             
database = "db"        
user = "user"             
password = "stuff"         
srid = 4326              
max_connections = 50  

  [[providers.layers]]
  #Layers and maps

During these caching runs with this updated setup, we have noticed intermittent errors when attempting to add to the s3 buckets:

Error: error seeding tile ({Z:14 X:4546 Y:6933}): RequestError: send request failed
caused by: Put "https://CACHEBUCKETNAME.s3.amazonaws.com/layers/14/4546/6933": write tcp IPADDRESS_1:50138->IPADDRESS_2:443: use of closed network connection

It would occur on different tiles, in different zoom levels. I observed it happening in approximately 10% of my processes (I parallelize the caching, running different geographical areas at the same time using AWS fargate instances).

Additionally, We noticed a similar issue on the postgis_mvt runs.

Error log:

Error: error seeding tile ({Z:16 X:14050 Y:27083}): InternalError: We encountered an internal error. Please try again.
	status code: 500, request id: , host id: 

Research indicates that the InternalError could be re-tried
https://aws.amazon.com/premiumsupport/knowledge-center/http-5xx-errors-s3/

@iwpnd
Copy link
Member

iwpnd commented Aug 18, 2022

hey @btwiningfc 👋

Given that this can be a result of too fast concurrent requests, I'd take it as a compliment for Tegola and the upgrade. 😄

Maybe it's worth a try to reduce parallelization, given the limits per prefix, and see whether you can bring that 10% down. If that is the root cause after all, there's no real change in Tegola necessary, tho not out of the equation either, but I leave that to @ARolek.
Just a general question out of interest, what do you see as a main benefit in seeding versus caching on request? Or in your case a mix of both.

Anyhows, I will try to take a look into a retry pattern, but no promises. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants