Assign specific jobs to dedicated workers #564

temaniarpit27 · 2024-08-29T07:03:43Z

#507

Cargo.toml

zero_bin/README.md

zero_bin/leader/src/main.rs

zero_bin/tools/prove_rpc.sh

zero_bin/leader/src/client.rs

zero_bin/leader/src/http.rs

zero_bin/leader/src/cli.rs

zero_bin/leader/src/main.rs

muursh

How have we tested this? Have we tested edge cases? Error cases?

temaniarpit27 · 2024-09-04T12:21:14Z

How have we tested this? Have we tested edge cases? Error cases?

Tested this on single and multi node machine. Tried to cover all the cases. Not sure if you have anything specific in mind
Started multiple workers on multiple nodes. Let me know if you have anything in mind and see if i have covered that

temaniarpit27 · 2024-09-05T06:37:43Z

@muursh One thing we might to check is when queue messages expire.

muursh · 2024-09-05T10:03:16Z

@muursh One thing we might to check is when queue messages expire.

Yeah for sure. We should also open issues for the bits we discussed.

temaniarpit27 · 2024-09-06T18:42:11Z

@muursh One thing we might to check is when queue messages expire.

Yeah for sure. We should also open issues for the bits we discussed.

Have asked leo to also confirm on a few test cases

zero_bin/prover/src/lib.rs

atanmarko

Lots of changes in the develop zero_bin project structure, will take a look one more time after rebase of this pr on top of develop

zero_bin/leader/src/main.rs

atanmarko · 2024-09-19T10:32:50Z

Cargo.toml

@@ -78,7 +78,7 @@ num-bigint = "0.4.5"
 num-traits = "0.2.19"
 nunny = "0.2.1"
 once_cell = "1.19.0"
-paladin-core = "0.4.2"
+paladin-core = { git = "https://github.com/0xPolygonZero/paladin.git", branch = "main" }


@muursh Are we going to release new paladin version or we are targeting github head?

afaik we will be releasing a new version

Yeah we should release a new version. @temaniarpit27 is everything ready to go for a new release?

Did we get actual benchmark measurements of this approach?

no we haven't benchmaked it yet

zero_bin/common/src/proof_runtime.rs

zero/src/bin/leader.rs

temaniarpit27 · 2024-10-03T12:12:22Z

@Nashtare final review from your side pls

Nashtare · 2024-10-03T12:14:27Z

zero/src/bin/leader/cli.rs

+    // Mode to use for worker for setup (affinity or default)
+    #[arg(long = "worker-run-mode", help_heading = WORKER_HELP_HEADING, value_enum, default_value = "default")]


what's default here? unified?

Nashtare · 2024-10-04T18:06:57Z

zero/src/prover.rs

+    pub light_proof_runtime: Runtime,
+    pub heavy_proof_runtime: Runtime,


minor, but I'd remove _runtime from the field names, as this is already implied by the parent

Nashtare · 2024-10-04T18:07:35Z

zero/src/prover.rs

@@ -28,6 +32,11 @@ use crate::fs::generate_block_proof_file_name;
 use crate::ops;
 use crate::proof_types::GeneratedBlockProof;

+pub struct ProofRuntime {


Could be nice to have some high-level explanation of this struct, i.e. its purpose to distinguish base STARK proof runtimes vs aggregation ones.

Nashtare · 2024-10-04T18:08:10Z

zero/src/bin/leader/cli.rs

+pub enum WorkerRunMode {
+    Affinity,
+    Default,
 }


Missing doc, we don't really know what this does just looking at it

separate workers

f0f4bf9

github-actions bot added the crate: zero_bin Anything related to the zero-bin subcrates. label Aug 29, 2024

temaniarpit27 commented Aug 29, 2024

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

temaniarpit27 commented Aug 29, 2024

View reviewed changes

zero_bin/README.md Outdated Show resolved Hide resolved

temaniarpit27 commented Aug 29, 2024

View reviewed changes

zero_bin/leader/src/main.rs Outdated Show resolved Hide resolved

temaniarpit27 commented Aug 29, 2024

View reviewed changes

zero_bin/leader/src/main.rs Outdated Show resolved Hide resolved

temaniarpit27 commented Aug 29, 2024

View reviewed changes

zero_bin/leader/src/main.rs Outdated Show resolved Hide resolved

temaniarpit27 changed the title ~~separate workers and jobs~~ Assign specific jobs to dedicated workers Aug 29, 2024

fix review comments

ee50295

temaniarpit27 force-pushed the arpit/507 branch from 5dff3b3 to ee50295 Compare August 30, 2024 11:56

Nashtare assigned temaniarpit27 Sep 1, 2024

Nashtare added this to the Performance Tuning milestone Sep 1, 2024

temaniarpit27 commented Sep 2, 2024

View reviewed changes

zero_bin/tools/prove_rpc.sh Outdated Show resolved Hide resolved

temaniarpit27 commented Sep 2, 2024

View reviewed changes

zero_bin/tools/prove_rpc.sh Outdated Show resolved Hide resolved

add worker run modes

e941b56

temaniarpit27 force-pushed the arpit/507 branch from d0b5d83 to e941b56 Compare September 2, 2024 08:41

temaniarpit27 added 2 commits September 2, 2024 18:07

change default queue

45cfeeb

update cargo.lock

8b00d3f

atanmarko reviewed Sep 2, 2024

View reviewed changes

zero_bin/tools/prove_rpc.sh Outdated Show resolved Hide resolved

zero_bin/tools/prove_rpc.sh Outdated Show resolved Hide resolved

zero_bin/leader/src/client.rs Outdated Show resolved Hide resolved

zero_bin/leader/src/http.rs Outdated Show resolved Hide resolved

temaniarpit27 added 2 commits September 3, 2024 17:13

fix review comments

a994162

merge develop

2e8f869

temaniarpit27 marked this pull request as ready for review September 3, 2024 13:14

temaniarpit27 requested review from muursh and Nashtare as code owners September 3, 2024 13:14

muursh reviewed Sep 4, 2024

View reviewed changes

zero_bin/leader/src/cli.rs Outdated Show resolved Hide resolved

muursh reviewed Sep 4, 2024

View reviewed changes

zero_bin/leader/src/main.rs Outdated Show resolved Hide resolved

muursh reviewed Sep 4, 2024

View reviewed changes

zero_bin/leader/src/main.rs Outdated Show resolved Hide resolved

muursh reviewed Sep 4, 2024

View reviewed changes

zero_bin/leader/src/main.rs Outdated Show resolved Hide resolved

muursh reviewed Sep 4, 2024

View reviewed changes

merge develop

af036b7

testing changes

3701b1e

temaniarpit27 force-pushed the arpit/507 branch from c1b19e4 to e0231ad Compare September 10, 2024 16:22

separate segment proof and segment agg proof

c410daa

temaniarpit27 force-pushed the arpit/507 branch from e0231ad to c410daa Compare September 10, 2024 16:22

temaniarpit27 commented Sep 10, 2024

View reviewed changes

zero_bin/prover/src/lib.rs Outdated Show resolved Hide resolved

merge develop

fb687b2

atanmarko reviewed Sep 12, 2024

View reviewed changes

zero_bin/prover/src/lib.rs Outdated Show resolved Hide resolved

temaniarpit27 added 2 commits September 13, 2024 18:53

review comments

fab7935

fix review comments

7d58fdf

temaniarpit27 force-pushed the arpit/507 branch from 7aaa0e6 to 7d58fdf Compare September 15, 2024 09:05

atanmarko reviewed Sep 19, 2024

View reviewed changes

temaniarpit27 force-pushed the arpit/507 branch 2 times, most recently from bba1ecb to 276a259 Compare September 19, 2024 12:38

atanmarko reviewed Sep 19, 2024

View reviewed changes

zero/src/bin/leader.rs Outdated Show resolved Hide resolved

merge develop

a56887b

temaniarpit27 force-pushed the arpit/507 branch from 276a259 to a56887b Compare September 19, 2024 12:45

atanmarko approved these changes Sep 19, 2024

View reviewed changes

merge develop

285b164

0xaatif mentioned this pull request Sep 26, 2024

pin toolchain #660

Closed

Merge branch 'develop' of github.com:0xPolygonZero/zk_evm into arpit/507

b8ee941

Nashtare reviewed Oct 4, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Assign specific jobs to dedicated workers #564

Assign specific jobs to dedicated workers #564

temaniarpit27 commented Aug 29, 2024

muursh left a comment

temaniarpit27 commented Sep 4, 2024

temaniarpit27 commented Sep 5, 2024

muursh commented Sep 5, 2024

temaniarpit27 commented Sep 6, 2024

atanmarko left a comment

atanmarko Sep 19, 2024

temaniarpit27 Sep 19, 2024

muursh Sep 19, 2024

Nashtare Sep 19, 2024

temaniarpit27 Sep 19, 2024 •

edited

Loading

temaniarpit27 commented Oct 3, 2024

Nashtare Oct 3, 2024

Nashtare Oct 4, 2024

Nashtare Oct 4, 2024

Nashtare Oct 4, 2024

		// Mode to use for worker for setup (affinity or default)
		#[arg(long = "worker-run-mode", help_heading = WORKER_HELP_HEADING, value_enum, default_value = "default")]

		pub light_proof_runtime: Runtime,
		pub heavy_proof_runtime: Runtime,

Assign specific jobs to dedicated workers #564

Are you sure you want to change the base?

Assign specific jobs to dedicated workers #564

Conversation

temaniarpit27 commented Aug 29, 2024

muursh left a comment

Choose a reason for hiding this comment

temaniarpit27 commented Sep 4, 2024

temaniarpit27 commented Sep 5, 2024

muursh commented Sep 5, 2024

temaniarpit27 commented Sep 6, 2024

atanmarko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

temaniarpit27 Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

temaniarpit27 commented Oct 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

temaniarpit27 Sep 19, 2024 •

edited

Loading