Issue with URL encoding of parameters in WMS

[ThomasG77]

Why? See below two URLs

Unworking URL. Obtained with Owslib that encode characters /,, et : and it's normal

http://ogc.geo-ide.developpement-durable.gouv.fr/wxs?map=/opt/data/carto/geoide-catalogue/1.4/org_38178/908a2fc2-6752-4eae-952a-142393e657b7.internet.map&service=WMS&version=1.3.0&request=GetMap&layers=N_PERIM_MAET_ZINF_S_R11&styles=&width=800&height=448&crs=EPSG%3A4326&bbox=48.1107%2C1.44041%2C49.2484%2C3.56583&format=image%2Fpng&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=XML

vs

Working URL

http://ogc.geo-ide.developpement-durable.gouv.fr/wxs?map=/opt/data/carto/geoide-catalogue/1.4/org_38178/908a2fc2-6752-4eae-952a-142393e657b7.internet.map&service=WMS&version=1.3.0&request=GetMap&layers=N_PERIM_MAET_ZINF_S_R11&styles=&width=800&height=448&crs=EPSG:4326&bbox=48.1107,1.44041,49.2484,3.56583&format=image/png&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=XML

The issue seems to be on the server side of the service I consume that does not decode the encoded parameters.

Code to reproduce how I got the following URL

from owslib.wms import WebMapService

wms = WebMapService('https://ogc.geo-ide.developpement-durable.gouv.fr/wxs?map=/opt/data/carto/geoide-catalogue/1.4/org_38178/908a2fc2-6752-4eae-952a-142393e657b7.internet.map', version='1.3.0')

response = wms.getmap(layers=['N_PERIM_MAET_ZINF_S_R11',],
    bbox=(1.44041, 48.1107, 3.56583, 49.2484),
    format='image/png',
    size=(800, 428),
    srs='EPSG:4326',
)
with open(f"N_PERIM_MAET_ZINF_S_R11.png", 'wb') as out:
    out.write(response.read())

Possible approach

To solve the issue, I want to use the safe option of https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlencode

getmap method would get a new safe option, empty by default (like the normal urlencode behavior) but could be changed at https://github.com/geopython/OWSLib/blob/master/owslib/map/wms111.py#L258 and https://github.com/geopython/OWSLib/blob/master/owslib/map/wms111.py#L258 and https://github.com/geopython/OWSLib/blob/master/owslib/map/wms130.py#L304
where

data = urlencode(request)

would be replaced by

data = urlencode(request,safe)

So, it can help solve the issue when you do not control the server encode/decode behavior while not breaking anything. I would be able to provide to the new safe option the value :/, in my case (already tested with hardcoded value while debugging) .

PS: I can do a PR but need to confirm the logic could be fine on owslib project. Moreover, I've only look at the WMS code part but this could be applied maybe to other types of webservices with the same issue.

[sbrunner]

When you create the WebMapService the map path should be encoded:

wms = WebMapService('https://ogc.geo-ide.developpement-durable.gouv.fr/wxs?map=%2Fopt%2Fdata%2Fcarto%2Fgeoide-catalogue%2F1.4%2Forg_38178%2F908a2fc2-6752-4eae-952a-142393e657b7.internet.map', version='1.3.0')

And you should be directly in https because the http to https redirect will break the URL...

[ThomasG77]

From a correct code perspective, the suggestion is surely right but it does not solve my sample code at all as it gives the same issue after change.

Did I miss something obvious?!

Ok. I understand that changing http to https in the samples URLs solve the issue but it's owlib that return/do http calls instead of https and I can't change the behavior without fixing issue at owlib level.

From my understanding, the issue is related to the fact owslib internally use http scheme from xlink:href in the capabilities from https://ogc.geo-ide.developpement-durable.gouv.fr/wxs?map=/opt/data/carto/geoide-catalogue/1.4/org_38178/908a2fc2-6752-4eae-952a-142393e657b7.internet.map&service=WMS&request=GetCapabilities so although the redirection http to https create "garbage", I do not see how a more obvious way to solve the issue as my first proposition.

[sbrunner]

With the version 0.25.0 it's working...

[sbrunner]

I think that a better solution is to have an option to skip this kind of code...

OWSLib/owslib/map/wms130.py

Lines 283 to 285 in 018c7f8

	base_url = next((m.get('url') for m in
	self.getOperationByName('GetMap').methods if
	m.get('type').lower() == method.lower()))

and use directly self.url :-)

[ThomasG77]

Using 0.27.2 and not working. Tested also with 0.26.0 and breaking too.

I've tested 0.25.0 and working with both cases e.g map url part encoded and unencoded...
Can you confirm/infirm regression?

Diagnostic: side effect due to another fix e.g

b3acd1b#diff-a8ea809e6cb216f4c6c765a0d19397bc6d463c883e37e16d9f4a08db87f2cdedL18-L200

Tested that changing back owslib/util.py in latest code solve the issue but will establish again the issue fixed on the WFS side (the cause of the code breaking change)

[sbrunner]

Finally, when I see the full code this looks relay strange

OWSLib/owslib/map/wms130.py

Line 304 in 018c7f8

data = urlencode(request)

The result is given to openURL, used here:
https://github.com/geopython/OWSLib/blob/018c7f83fb042a6390d65aaa683110743609f3e3/owslib/util.py#LL200C33-L200C33
and given to requests here:

OWSLib/owslib/util.py

Line 208 in 018c7f8

req = requests.request(method.upper(), url_base, headers=headers, **rkwargs)

In requests documentation, we see that prams should be a dictionary
https://tedboy.github.io/requests/generated/generated/requests.Request.html
And there is a duplication
Then this code

OWSLib/owslib/map/wms130.py

Line 308 in 018c7f8

u = openURL(base_url, data, method, timeout=timeout or self.timeout, auth=self.auth, headers=self.headers)

should be:

        u = openURL(base_url, request, method, timeout=timeout or self.timeout, auth=self.auth, headers=self.headers)

The same issue is in many place in the code...

[yagossc]

Is this still a problem?

I'm using __version__ = '0.31.0'

I create a wfs instance passing a URL that uses HTTPS:

WebFeatureService("https://domain/path/etc", version="2.0.0")

However, when I call the method getfeature(typename="blabla", ...) the request is made using HTTP.

The server I'm trying to reach only allows https requests and returns redirects otherwise (i.e., 302 Found).

Is there a workaround somewhere that I couldn't find? This is the only issue I found mentioning this specific case.