[R] Discussion: libarrow backwards compatibility enforcement

[jonkeane]

Describe the enhancement requested

For a while we were able to support older versions of libarrow, but since then we have had a few PRs bump into this and had to either bump our minimum version or add extra code around this (#41998 and #42241 are ones I an think of off the top of my head). In both circumstances there was some discussion / friction about what the policy is, if we must retain backwards compatibility, and if so what the best way is to approach that.

It's my understanding that there is no current live user(s) of using the backwards compatibility and especially no specific "I have libarrow 15, but need current versions of the arrow R package" use cases. Because of this, we are free (and should feel free) to increase the required minimum version as we need to in order to introduce new features. But it seems like there is some ambiguity in this. I would love this issue to be a place of discussion around this for anyone who has opinions. I tried to see if we already had documentation explaining this + the philosophy here, but I didn't see any (though please correct me if I missed it!), but the outcome of this ideally is an update to our docs so our stance + recommendations on this are clear.

Component(s)

R

[amoeba]

Hey @jonkeane, thanks for starting this discussion. I'm pretty split on this but I'm leaning towards removing the CI job and not enforcing backwards-compatibility.

If we were to ever get libarrow on the CRAN check machines, we would then have to contend with whatever version of libarrow CRAN was on and it will likely be less up to date than we'd like. Having a well-stated goal of maximizing the number of previous major libarrow versions we can build the R package with would help us in this situation.

However, (1) the above scenario isn't very likely to happen and (2) the backwards compatibility job has caused friction and confusion, most recently with #44357. In the latter and other cases, I think part of the confusion is due to the fact that no other components of the Arrow project have anything similar. It would be less confusing if, for example, PyArrow had a similar policy.

I'd like to hear others' thoughts as well.