bug(ci): multi-version tikv-jemalloc-sys break docker pipeline #12315

MrCroxx · 2023-09-14T11:38:59Z

Describe the bug

RisingWave got 2 version of tikv-jemalloc-sys built in docker pipeline. And it breaks CI.

cp ./target/release/build/tikv-jemalloc-sys-*/out/build/bin/jeprof /risingwave/bin/

Error message/log

<img width="1055" alt="image" src="https://github.com/risingwavelabs/risingwave/assets/22407295/21c68b59-7d1b-47a8-82a9-df8f30f9bb03">

The text was updated successfully, but these errors were encountered:

MrCroxx · 2023-09-14T11:40:41Z

Thanks to @xxchan for helping. The bug might be introduced by #12276 .

UPDATE: It's caused by #12307

MrCroxx · 2023-09-14T11:43:09Z

cargo tree -d got:

[build-dependencies]
├── foyer-workspace-hack v0.1.0 (https://github.com/mrcroxx/foyer?rev=41b1d39#41b1d393) (*)
└── workspace-hack v1.1.0-alpha (/home/mrcroxx/github/risingwavelabs/risingwave/src/workspace-hack) (*)

tikv-jemalloc-sys v0.5.4+5.3.0-patched (https://github.com/risingwavelabs/jemallocator.git?rev=64a2d9#64a2d988)
[build-dependencies]
└── workspace-hack v1.1.0-alpha (/home/mrcroxx/github/risingwavelabs/risingwave/src/workspace-hack) (*)

tikv-jemalloc-sys v0.5.4+5.3.0-patched (https://github.com/risingwavelabs/jemallocator.git?rev=64a2d9#64a2d988)
├── tikv-jemalloc-ctl v0.5.4 (https://github.com/risingwavelabs/jemallocator.git?rev=64a2d9#64a2d988)
│   └── risingwave_compute v1.1.0-alpha (/home/mrcroxx/github/risingwavelabs/risingwave/src/compute) (*)
├── tikv-jemallocator v0.5.4 (https://github.com/risingwavelabs/jemallocator.git?rev=64a2d9#64a2d988)
│   ├── risingwave_cmd v1.1.0-alpha (/home/mrcroxx/github/risingwavelabs/risingwave/src/cmd) (*)
│   ├── risingwave_cmd_all v1.1.0-alpha (/home/mrcroxx/github/risingwavelabs/risingwave/src/cmd_all)
│   └── risingwave_simulation v0.1.0 (/home/mrcroxx/github/risingwavelabs/risingwave/src/tests/simulation)
│   [dev-dependencies]
│   └── risingwave_batch v1.1.0-alpha (/home/mrcroxx/github/risingwavelabs/risingwave/src/batch) (*)
└── workspace-hack v1.1.0-alpha (/home/mrcroxx/github/risingwavelabs/risingwave/src/workspace-hack) (*)

And compared with cargo tree -d, the two non-duplicated tikv-jemalloc-sys dependency comes from RisingWave's dependency and build-dependency.

MrCroxx · 2023-09-14T11:57:33Z

#12307

xxchan · 2023-09-15T03:11:06Z

The duplicated versions are with and without feature unprefixed_malloc_on_supported_platforms, which is target.'cfg(unix)'.dependencies.

I guess it's because the build-dependency (introduced by hakari) doesn't contain this feature.

We can see cargo does not unify features for dependencies and build-dependencies

[dependencies]
rand = { version = "0.8.5", default-features = false }

[build-dependencies]
rand = { version = "0.8.5", default-features = false, features = ["serde"] }


❯ cargo tree -p rand -e features -i              
rand v0.8.5
└── tmp v0.1.0 (/private/tmp/tmp)
    └── tmp feature "default" (command-line)

rand v0.8.5
└── rand feature "serde"
    [build-dependencies]
    └── tmp v0.1.0 (/private/tmp/tmp) (*)

xxchan · 2023-09-15T03:14:02Z

If we add target in hakari config

risingwave/.config/hakari.toml

Line 18 in 214118b

# "x86_64-unknown-linux-gnu",

It will add new section

[target.x86_64-unknown-linux-gnu.dependencies]
#...
tikv-jemalloc-sys = { git = "https://github.com/risingwavelabs/jemallocator.git", rev = "64a2d9", default-features = false, features = ["unprefixed_malloc_on_supported_platforms"] }
tikv-jemallocator = { git = "https://github.com/risingwavelabs/jemallocator.git", rev = "64a2d9", features = ["profiling", "stats", "unprefixed_malloc_on_supported_platforms"] }
#...

[target.x86_64-unknown-linux-gnu.build-dependencies]
#...
tikv-jemalloc-sys = { git = "https://github.com/risingwavelabs/jemallocator.git", rev = "64a2d9", default-features = false, features = ["unprefixed_malloc_on_supported_platforms"] }
tikv-jemallocator = { git = "https://github.com/risingwavelabs/jemallocator.git", rev = "64a2d9", features = ["profiling", "stats", "unprefixed_malloc_on_supported_platforms"] }
#...

xxchan · 2023-09-15T03:26:16Z

Alternatively, this also exclude it from hakari

diff --git a/.config/hakari.toml b/.config/hakari.toml
index e998f5fea..9cdfbb3b0 100644
--- a/.config/hakari.toml
+++ b/.config/hakari.toml
@@ -35,6 +35,4 @@ third-party = [
     { name = "criterion" },
     { name = "console" },
     { name = "similar" },
-    # FYI: https://github.com/risingwavelabs/risingwave/issues/12315
-    { name = "tikv-jemalloc-sys", git = "https://github.com/risingwavelabs/jemallocator.git", rev = "64a2d9" },
 ]
diff --git a/src/compute/Cargo.toml b/src/compute/Cargo.toml
index 24d12151c..6e3f8674a 100644
--- a/src/compute/Cargo.toml
+++ b/src/compute/Cargo.toml
@@ -39,7 +39,6 @@ risingwave_storage = { workspace = true }
 risingwave_stream = { workspace = true }
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"
-tikv-jemalloc-ctl = { git = "https://github.com/risingwavelabs/jemallocator.git", rev = "64a2d9" }
 tokio = { version = "0.2", package = "madsim-tokio", features = [
     "rt",
     "rt-multi-thread",
@@ -54,6 +53,10 @@ tonic = { workspace = true }
 tower = { version = "0.4", features = ["util", "load-shed"] }
 tracing = "0.1"
 
+[target.'cfg(unix)'.dev-dependencies]
+tikv-jemalloc-ctl = { git = "https://github.com/risingwavelabs/jemallocator.git", rev = "64a2d9" }
+
+
 [target.'cfg(not(madsim))'.dependencies]
 workspace-hack = { path = "../workspace-hack" }
 
diff --git a/src/tests/simulation/Cargo.toml b/src/tests/simulation/Cargo.toml
index 82992b8b0..01c20b172 100644
--- a/src/tests/simulation/Cargo.toml
+++ b/src/tests/simulation/Cargo.toml
@@ -46,11 +46,14 @@ serde_derive = "1.0.188"
 serde_json = "1.0.107"
 sqllogictest = "0.15.3"
 tempfile = "3"
-tikv-jemallocator = { workspace = true }
 tokio = { version = "0.2.23", package = "madsim-tokio" }
 tokio-postgres = "0.7"
 tracing = "0.1"
 tracing-subscriber = { version = "0.3", features = ["env-filter"] }
 
+[target.'cfg(unix)'.dev-dependencies]
+tikv-jemallocator = { workspace = true }
+
+
 [lints]
 workspace = true

xxchan · 2023-09-15T03:27:07Z

By default, cargo hakari produces the minimal set of features that can be unified across all possible platforms. However, in practice, most developers on a codebase use one of a few platforms. cargo hakari can run specific queries for a few platforms, producing better results for them.

MrCroxx · 2023-09-15T07:25:56Z

@xxchan Seems we can finally close this issue via #12333 ?

MrCroxx · 2023-09-15T11:47:56Z

[target.x86_64-unknown-linux-gnu.dependencies]
axum = { version = "0.6" }
memchr = { version = "2" }
mime_guess = { version = "2" }
miniz_oxide = { version = "0.7", default-features = false, features = ["with-alloc"] }
once_cell = { version = "1", features = ["unstable"] }
openssl-sys = { version = "0.9", default-features = false, features = ["vendored"] }
opentelemetry = { version = "0.20", default-features = false, features = ["metrics", "rt-tokio", "trace"] }
opentelemetry_sdk = { version = "0.20", default-features = false, features = ["rt-tokio"] }
rdkafka-sys = { git = "https://github.com/MaterializeInc/rust-rdkafka", rev = "8ea07c4", default-features = false, features = ["cmake-build", "gssapi", "libz", "ssl-vendored", "zstd"] }
rustix = { version = "0.38", features = ["fs", "termios"] }
rustls = { version = "0.21", features = ["dangerous_configuration"] }
serde_json = { version = "1", default-features = false, features = ["raw_value"] }
tikv-jemalloc-sys = { git = "https://github.com/risingwavelabs/jemallocator.git", rev = "64a2d9", features = ["profiling", "stats", "unprefixed_malloc_on_supported_platforms"] }
tikv-jemallocator = { git = "https://github.com/risingwavelabs/jemallocator.git", rev = "64a2d9", features = ["profiling", "stats", "unprefixed_malloc_on_supported_platforms"] }
tower-http = { version = "0.4", features = ["add-extension", "cors", "fs"] }
zstd-sys = { version = "2", features = ["std"] }

[target.x86_64-unknown-linux-gnu.build-dependencies]
axum = { version = "0.6" }
memchr = { version = "2" }
mime_guess = { version = "2" }
miniz_oxide = { version = "0.7", default-features = false, features = ["with-alloc"] }
once_cell = { version = "1", features = ["unstable"] }
openssl-sys = { version = "0.9", default-features = false, features = ["vendored"] }
opentelemetry = { version = "0.20", default-features = false, features = ["metrics", "rt-tokio", "trace"] }
opentelemetry_sdk = { version = "0.20", default-features = false, features = ["rt-tokio"] }
rdkafka-sys = { git = "https://github.com/MaterializeInc/rust-rdkafka", rev = "8ea07c4", default-features = false, features = ["cmake-build", "gssapi", "libz", "ssl-vendored", "zstd"] }
rustix = { version = "0.38", features = ["fs", "termios"] }
rustls = { version = "0.21", features = ["dangerous_configuration"] }
serde_json = { version = "1", default-features = false, features = ["raw_value"] }
tikv-jemalloc-sys = { git = "https://github.com/risingwavelabs/jemallocator.git", rev = "64a2d9", features = ["profiling", "stats", "unprefixed_malloc_on_supported_platforms"] }
tikv-jemallocator = { git = "https://github.com/risingwavelabs/jemallocator.git", rev = "64a2d9", features = ["profiling", "stats", "unprefixed_malloc_on_supported_platforms"] }
tower-http = { version = "0.4", features = ["add-extension", "cors", "fs"] }
zstd-sys = { version = "2", features = ["std"] }

@xxchan tikv-jemalloc-sys duplicated again. 🥹

MrCroxx · 2023-09-15T12:13:12Z

And this time both tikv-jemallocator and tikv-jemalloc-sys are listed. 😭

xxchan · 2023-09-15T12:59:28Z

❯ cargo tree -d  | rg tikv

There are no duplicates

MrCroxx · 2023-09-15T13:15:07Z

❯ cargo tree -d  | rg tikv   
There are no duplicates

Wired. But it did build 2 duplicated targets. https://buildkite.com/risingwavelabs/docker/builds/14287

xxchan · 2023-09-17T00:56:28Z

I found debug build doesn't have duplicates, but release build has. Too weird..

xxchan · 2023-09-17T01:07:07Z

I found one copy is release build, while another is debug build 🤯

xxchan · 2023-09-17T01:55:54Z

I think this is because (dependency's) build-dependency is built with different config.

xxchan · 2023-09-17T01:57:55Z

https://doc.rust-lang.org/cargo/reference/profiles.html#build-dependencies

To compile quickly, all profiles, by default, do not optimize build dependencies

xxchan · 2023-09-17T02:33:28Z

guppy-rs/guppy#83

Before: `cargo build --release`: Building [========> ] ../1339 After: Building [========> ] ../1065 Background #12315 (comment)

fuyufjh · 2023-10-05T09:26:10Z

This doesn't seem to be caused by [build-dependencies]

In my cases (#12625), tikv-jemalloc-sys was produced under [dependencies] but it still compiles both for release and debug, exactly as #12315 (comment) shows.

MrCroxx added the type/bug Something isn't working label Sep 14, 2023

github-actions bot added this to the release-1.3 milestone Sep 14, 2023

This was referenced Sep 14, 2023

fix(ci): build jemallocator only on linux #12316

Closed

fix(ci): exclude tikv-jemalloc-sys in hakari check #12320

Merged

xxchan mentioned this issue Sep 15, 2023

chore: add platforms to hakari #12333

Merged

8 tasks

xxchan added a commit that referenced this issue Sep 17, 2023

chore: avoid [build-dependencies] being compiled twice

b96fda9

Before: `cargo build --release`: Building [========> ] ../1339 After: Building [========> ] ../1065 Background #12315 (comment)

This was referenced Sep 17, 2023

chore: avoid [build-dependencies] being compiled twice #12362

Closed

revert: Revert "chore: add platforms to hakari (#12333)" #12363

Merged

xxchan mentioned this issue Oct 5, 2023

fix: enable jemalloc-based memory control on macOS as well #12625

Merged

8 tasks

fuyufjh closed this as completed Oct 10, 2023

xxchan mentioned this issue Oct 19, 2023

build: remove workspace-hack from proc-macro crates' dependencies #12961

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bug(ci): multi-version tikv-jemalloc-sys break docker pipeline #12315

bug(ci): multi-version tikv-jemalloc-sys break docker pipeline #12315

MrCroxx commented Sep 14, 2023 •

edited

Loading

MrCroxx commented Sep 14, 2023 •

edited

Loading

MrCroxx commented Sep 14, 2023 •

edited

Loading

MrCroxx commented Sep 14, 2023

xxchan commented Sep 15, 2023

xxchan commented Sep 15, 2023 •

edited

Loading

xxchan commented Sep 15, 2023

xxchan commented Sep 15, 2023

MrCroxx commented Sep 15, 2023

MrCroxx commented Sep 15, 2023 •

edited

Loading

MrCroxx commented Sep 15, 2023

xxchan commented Sep 15, 2023

MrCroxx commented Sep 15, 2023

xxchan commented Sep 17, 2023

xxchan commented Sep 17, 2023

xxchan commented Sep 17, 2023

xxchan commented Sep 17, 2023

xxchan commented Sep 17, 2023

fuyufjh commented Oct 5, 2023

bug(ci): multi-version tikv-jemalloc-sys break docker pipeline #12315

bug(ci): multi-version tikv-jemalloc-sys break docker pipeline #12315

Comments

MrCroxx commented Sep 14, 2023 • edited Loading

Describe the bug

Error message/log

MrCroxx commented Sep 14, 2023 • edited Loading

MrCroxx commented Sep 14, 2023 • edited Loading

MrCroxx commented Sep 14, 2023

xxchan commented Sep 15, 2023

xxchan commented Sep 15, 2023 • edited Loading

xxchan commented Sep 15, 2023

xxchan commented Sep 15, 2023

MrCroxx commented Sep 15, 2023

MrCroxx commented Sep 15, 2023 • edited Loading

MrCroxx commented Sep 15, 2023

xxchan commented Sep 15, 2023

MrCroxx commented Sep 15, 2023

xxchan commented Sep 17, 2023

xxchan commented Sep 17, 2023

xxchan commented Sep 17, 2023

xxchan commented Sep 17, 2023

xxchan commented Sep 17, 2023

fuyufjh commented Oct 5, 2023

MrCroxx commented Sep 14, 2023 •

edited

Loading

MrCroxx commented Sep 14, 2023 •

edited

Loading

MrCroxx commented Sep 14, 2023 •

edited

Loading

xxchan commented Sep 15, 2023 •

edited

Loading

MrCroxx commented Sep 15, 2023 •

edited

Loading