Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Next round of abseil / grpc / protobuf upgrades #4075

Open
3 tasks done
h-vetinari opened this issue Feb 8, 2023 · 38 comments · Fixed by #4482
Open
3 tasks done

Next round of abseil / grpc / protobuf upgrades #4075

h-vetinari opened this issue Feb 8, 2023 · 38 comments · Fixed by #4482

Comments

@h-vetinari
Copy link
Member

h-vetinari commented Feb 8, 2023

It's that time of the year again, it seems. We have:

in the pipeline. The first two should be straight-forward, though protobuf this time breaks the C++ API (a little), and adds a dependency on (the newest) abseil. Since wait_for_migrators: isn't working currently, I'd like to explicitly order the migrations as above, i.e. first abseil, then grpc, then protobuf.

This is also somewhat related to #4068, though I don't (yet) see that we'd need to build libprotobuf twice.

CC @conda-forge/abseil-cpp @conda-forge/grpc-cpp @conda-forge/libprotobuf

PS. The first two are also dropping the compatibility outputs for the respective old names (abseil-cpp / grpc-cpp); hence we need to exclude them from the migrator, but I'd suggest to keep them in the pinning with the last available version, at least for a while longer.

@coryan
Copy link

coryan commented Feb 8, 2023

FYI, gRPC v1.52.0 does not build with libprotobuf 4.22. Why? Because in protocolbuffers/protobuf@01fe222 some header files got renamed, and gRPC depends on the old names:

https://github.com/grpc/grpc/blob/v1.52.0/src/compiler/config_protobuf.h#L53

I have asked the gRPC team to support both protobuf-3.21 and protobuf-4.22 in their upcoming 1.53.0 release, but they don't know (yet) if it would be possible.

Is there any way to change the recipes to say "grpc <= 1.52 needs protobuf < 4.0 and grpc > 1.52 needs protobuf >= 4.0"?

@h-vetinari
Copy link
Member Author

Thanks a lot for the info, @coryan!

Is there any way to change the recipes to say "grpc <= 1.52 needs protobuf < 4.0 and grpc > 1.52 needs protobuf >= 4.0"?

Yes, that's possible, or rather, the default. If we don't build for a specific combination, it will not be supported. We could backport a possible header rename to grpc 1.52 if it's not too invasive, but otherwise it sounds like we might have to wait with protobuf a bit.

@h-vetinari
Copy link
Member Author

h-vetinari commented Apr 19, 2023

Alrighty, we have a new grpc version (1.54; which should be compatible with protobuf 4) ready to be migrated: #4353. After that we should be good to go to crack 4.22.x & migrate it.

@h-vetinari
Copy link
Member Author

So conda-forge/libprotobuf-feedstock#144 was finally merged (long stuck on painful abseil preparations), but in contrast to the 21 cycle (which lasted from May 2022 until February 2023), we now already have protobuf 23 that's out three months after 22.x.

I was going to propose having a migrator that does a dual protobuf pin for 3.21 & 4.22 (along the lines of #4068), but now we'd either need three parallel protobuf versions (while we're waiting to digest the breaking changes on C++ side), or we skip the 22-series directly.

@coryan
Copy link

coryan commented May 13, 2023

or we skip the 22-series directly.

I would. No gRPC release compiles (or will compile) with the 22.x series. The gRPC-1.55.x series (due "any day now") will only compile with Protobuf == 23.x (and hopefully with Protobuf >= 23.x, but I need to confirm that).

@h-vetinari
Copy link
Member Author

Thanks for the update! Yeah, then I agree it looks like we should skip 22.x.

and hopefully with Protobuf >= 23.x, but I need to confirm that

It would be unfortunate IMO if grpc & protobuf were coupled this hard, but I guess we can deal with it.

@h-vetinari
Copy link
Member Author

h-vetinari commented May 13, 2023

Though the freshly updated recipe for libprotobuf 4.22 does not work for 4.23 anymore (C++ compilation errors), not sure how long it'll take to get it up to speed.

@h-vetinari
Copy link
Member Author

As an added wrinkle, the C++ backend for the python bindings is being deprecated, and the replacement is based on a library without any releases that constantly gets revendored in protobuf. See conda-forge/protobuf-feedstock#170

This was referenced May 14, 2023
@coryan
Copy link

coryan commented May 23, 2023

FWIW, Protobuf 23.1 (aka 4.23.1) and gRPC 1.55.0 compile together, yay! The next release of google-cloud-cpp will compile with them too.

@coryan
Copy link

coryan commented May 23, 2023

As an added wrinkle, the C++ backend for the python bindings is being deprecated, and the replacement is based on a library without any releases that constantly gets revendored in protobuf. See conda-forge/protobuf-feedstock#170

The wrinkle gets worse when you look at gRPC, which currently vendors in upb:

https://github.com/grpc/grpc/tree/master/third_party/upb

Moreover, upb generates C bindings and gRPC commits generated files to their repository:

https://github.com/grpc/grpc/tree/master/src/core/ext/upbdefs-generated/google/protobuf

And if that was not enough complexity, upb makes no guarantees about using the generated code with any version of upb other than the version used to generate it.

The teams involved are aware of how undesirable all this situation is, and have plans to fix it. But the fixes are not going to happen "soon".

@h-vetinari
Copy link
Member Author

h-vetinari commented May 23, 2023

The wrinkle gets worse when you look at gRPC, which currently vendors in upb:

Yeah, I'm aware... I've been wondering how to deal with this, but I first wanted to try how a build of protobuf-with-upb looks like (though I expect the python bindings will put upb under site-packages, whereas grpc puts its upb under $PREFIX/lib). In any case, it would be bad to have diverging upb artefacts on the path, so I would have very likely had to raise this upstream in any case.

So I'm very glad you could provide some further insights.

The teams involved are aware of how undesirable all this situation is, and have plans to fix it. But the fixes are not going to happen "soon".

One key aspect here would be to not remove the deprecated CPP backend of the python bindings, before the upb situation has been resolved.

@h-vetinari
Copy link
Member Author

h-vetinari commented May 24, 2023

So, after merging conda-forge/libprotobuf-feedstock#158, conda-forge/protobuf-feedstock#187 and (finally!) conda-forge/grpc-cpp-feedstock#243, we're IMO ready to migrate here: #4482

@h-vetinari
Copy link
Member Author

Could someone please reopen this issue? It seems we need a place to discuss issues that arise while migrating protobuf 4, and this one has all the context.

@jakirkham jakirkham reopened this May 25, 2023
@jakirkham
Copy link
Member

Ofc, reopened

@h-vetinari
Copy link
Member Author

In general this protobuf migration has the potential to be more painful than usual: every package now needs to depend on (and link to!) abseil, and compile with C++17 (to say nothing of potential source incompatibilities with the new major version). For packages where switching to C++17 might also bring an ABI change, we'll have to add them to the migrator.

@traversaro
Copy link
Contributor

traversaro commented May 25, 2023

First of all, thanks a lot @h-vetinari for all the handling of the protobuf situation!

compile with C++17

I probably know the answer, but just for people reading: the requirements for all compilation units that include (even transitively) abseil headers is to compile with the exact version of C++ used to compile abseil (so as of 2023/05 C++17), so it is not possible to use a newer version of C++ (C++20 or C++23), is that correct?

@h-vetinari
Copy link
Member Author

h-vetinari commented May 25, 2023

First of all, thanks a lot @h-vetinari for all the handling of the protobuf situation!

Thanks for the kind words!

the requirements for all compilation units that include (even transitively) abseil headers is to compile with the exact version of C++ used to compile abseil (so as of 2023/05 C++17), so it is not possible to use a newer version of C++ (C++20 or C++23), is that correct?

Historically we've gotten away with diverging standard versions between compilation units (especially on unix), though abseil does not consider this supported. It'll also generally fail loudly with linker errors if something is not compatible. However, C++17 as a baseline homogenizes many of the ABI-diverging abseil backports, and C++20 in abseil doesn't yet have major ABI differences, much less ones that get a lot of usage (certainly not on the scale of e.g. absl::string_view and absl::optional before C++17).

So strictly speaking, only C++17 is supported, but C++20 will very likely work1. That said, once abseil's ABI diverges between C++17 & C++20 (in a way that gets used in actual projects), we'll probably have to come back to fixing conda-forge/abseil-cpp-feedstock#45 (more details & references in that issue).

Footnotes

  1. by the same token C++23 should too, but I wouldn't recommend use of that in feedstocks; completely aside from distribution questions, support has not been finalized by any of the main compilers yet.

@h-vetinari
Copy link
Member Author

h-vetinari commented May 26, 2023

Ugh, this is getting nastier and nastier. It seems that grpc now depends on the exact patch version of protobuf used to build it. After conda-forge/libprotobuf-feedstock#159 was merged today, I'm seeing the following In conda-forge/cpp-opentelemetry-sdk-feedstock#48:

$PREFIX/bin/grpc_cpp_plugin: error while loading shared libraries: libprotoc.so.23.1.0: cannot open shared object file: No such file or directory
dyld: Library not loaded: @rpath/libprotoc.23.1.0.dylib
  Referenced from: $PREFIX/bin/grpc_cpp_plugin
  Reason: image not found

(though interestingly not on windows).

Unless we want to migrate for protobuf patch versions too (nope nope nope), I think we should consider patching out something here (whether the SOVER format in libprotoc, or grpc_cpp_plugin's dependence on it).

@traversaro
Copy link
Contributor

Perhaps the linking on the exact patch name is accidental? I have a vague remembering of seeing a similar issue in the past, where the build system was accidentally linking the library with the full version instead of the symlink with the soversion. Let me check this.

@h-vetinari
Copy link
Member Author

So I've had a look at the status of the migrator, and we're now pretty much only in leaf-feedstock territory, where people haven't looked at the respective PRs for weeks since they got opened.

I've just merged conda-forge/libprotobuf-feedstock#166 & conda-forge/grpc-cpp-feedstock#304, will have a PR up momentarily to (re)start a new migration for that.

One of the key changes is that now the PROTOBUF_USE_DLLS symbols is not necessary anymore on unix to get the correct ABI and find the google::protobuf::internal::ThreadSafeArena::thread_cache_ symbol (which previously changed depending on the presence/absence of that symbol).

@xhochy
Copy link
Member

xhochy commented Jun 30, 2023

I guess there isn't a particular version of tensorflow, but more the willingness to not bifurcate the conda-forge ecosystem

@hmaarrfk @ngam From what it seems, I got 2.12 with an up-to-date rerender building. I prefer to build them step-by-step as it would ease returning to solvable environments. I have the necessary CPU time available to do all these builds. Human time to spend on this is limited but overall (at least for me), it would help be less time to spend on issues if we had tensorflow 2.12 available for several pinnings.

@h-vetinari
Copy link
Member Author

Seems the new protobuf feature-versions are getting more and more short-lived. v24 is in rc2 now, and brings some big changes again, not least: protobuf editions (looks modeled after rust at first glance).

This looks like a good thing long-term though, and for now it's hidden behind an experimental flag. Still no news on the upb-situation (between grpc & protobuf).

@h-vetinari
Copy link
Member Author

Alright, seems like it's google release day:

I was actually waiting with conda-forge/libprotobuf-feedstock#168 until grpc 1.57 hit (because we've been coupling the migrations). I think that's still the best approach for now, as I expect no major hiccoughs from protobuf 4.23.{3->4} and grpc 1.{56->57}. I think protobuf 4.24 might take a while again, so it's not an issue to wait a bit.

Not sure if we want to couple the abseil migration with grpc/protobuf. I guess it would make sense (as long as they're not zipped together, because protobuf 3.x also needs abseil).

@h-vetinari
Copy link
Member Author

An update - after much struggling with conda-forge/protobuf-feedstock#215 due to upstream switching to bazel (as well as the main backend to upb, etc.) and a heroic effort by @xhochy to fix it, I've now started a new migration for the newest abseil, plus protobuf 5.27.5 & grpc 1.65.

If/once that goes through, we should hopefully be able to catch up again, and then stay up-to-date as before. There might still be some rough edges though. Please let us know if you run into something.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
7 participants