Feature/wdm next refactor #341

allenshihgoog · 2019-08-20T19:33:14Z

Major refactor improvements on wdmNext automation infrastructure.
This is PHASE 1 + PHASE 2 of the design

PHASE 1:

Refactor all parameter options and default values to const file WeaveWdmNextOptions.py.
Options now split between test/client/server
Naming of options now consistent with test-app CLI arguments.
Each parameter option is documented for its use case
Improved processing of client/server options into optimized loop reducing 130 lines -> 20 lines

PHASE 2:

Refactor ALL test parameters for 90+ test files into weave_wdm_next_test_params.json keyed by the test name and then test/client/server parameters. All test parameters were parsed using this script
All 90 tests now will read from json to run their test params.

Main files to review:

weave_wdm_next_test_base.py
WeaveWdmNext.py
WeaveWdmNextOptions.py
weave_wdm_next_test_params.json

All functions in these files are now documented on what they do, what their parameters are, and what they return.

PHASE 3 will be next to group test cases so that they can share the same setup and teardown of weave topology to significantly increase testing speed.

NOTE:
Changes for wdmNext service tests are NOT here. I need to work with @yunhanw-google to figure out how to run them.

googlebot · 2019-08-20T19:33:17Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it!) and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

googlebot · 2019-08-20T19:33:17Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it!) and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

gerickson · 2019-08-23T15:35:58Z

If this is the "base" of what becomes normal, is there a need to name the files as part of this "Next" and "next"? I'd recommend just dropping those naming decorations.

gerickson

At minimum, please see my naming feedback. Beyond that, I have a few minor comments in the design document.

codecov-io · 2019-08-27T01:06:34Z

Codecov Report

Merging #341 into master will decrease coverage by 1.7%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #341      +/-   ##
==========================================
- Coverage   54.46%   52.76%   -1.71%     
==========================================
  Files         340      326      -14     
  Lines       58425    56047    -2378     
==========================================
- Hits        31821    29573    -2248     
+ Misses      26604    26474     -130

Impacted Files	Coverage Δ
src/inet/InetFaultInjection.cpp	`0% <0%> (-100%)`	⬇️
src/test-apps/ToolCommon.h	`0% <0%> (-100%)`	⬇️
src/test-apps/TestInetEndPoint.cpp	`1.8% <0%> (-96.8%)`	⬇️
src/inet/InetUtils.cpp	`0% <0%> (-95.35%)`	⬇️
src/inet/InetInterface.cpp	`0% <0%> (-65.1%)`	⬇️
src/inet/InetLayer.cpp	`4.01% <0%> (-63.77%)`	⬇️
src/lib/profiles/security/WeaveAccessToken.cpp	`0% <0%> (-48.15%)`	⬇️
src/inet/DNSResolver.cpp	`0% <0%> (-47.56%)`	⬇️
src/inet/UDPEndPoint.cpp	`0% <0%> (-43.16%)`	⬇️
src/lib/profiles/security/WeaveCert.h	`61.53% <0%> (-38.47%)`	⬇️
... and 40 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a95ab81...ea7b7bc. Read the comment docs.

allenshihgoog · 2019-08-27T17:39:45Z

@googlebot I signed it!

googlebot · 2019-08-27T17:39:50Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

googlebot · 2019-08-27T17:39:50Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

src/test-apps/happy/test-templates/WeaveWdmNextOptions.py

src/test-apps/happy/tests/standalone/wdmNext/weave_wdm_next_test_base.py

src/test-apps/happy/test-templates/WeaveWdmNextOptions.py

robszewczyk · 2019-08-29T23:11:46Z

If this is the "base" of what becomes normal, is there a need to name the files as part of this "Next" and "next"? I'd recommend just dropping those naming decorations.

I would suggest we decouple concerns: lets refactor the tests and then rename them; at the moment, this PR still maintains the 1-1 correspondence between the previous tests and the new ones; for traceability purposes, it may be cleaner to have at least separate commits but separate PRs may be more appropriate given the size of the change.

allenshihgoog · 2019-09-17T16:43:45Z

Here are logs of Jenkins job using my branch to run the WDM service tests. Logs indicate the same results as running off of master branch that CSI team uses to pull data from.
Note: Not all of the tests "pass", but it seems CSI team pulls other data from the runs which is sufficient for them to consider it passing.

https://paste.googleplex.com/4976942140882944
Jenkins log link was removed because they run many times a day and don't keep older runs.

jenniexie requested review from robszewczyk, yunhanw-google and jenniexie August 20, 2019 22:16

allenshihgoog force-pushed the feature/wdmNextRefactor branch from c071e2b to 73d051c Compare August 22, 2019 17:39

gerickson suggested changes Aug 23, 2019

View reviewed changes

allenshihgoog force-pushed the feature/wdmNextRefactor branch 2 times, most recently from f978568 to 0f9f23c Compare August 27, 2019 00:11

jenniexie reviewed Aug 27, 2019

View reviewed changes

src/test-apps/happy/test-templates/WeaveWdmNextOptions.py Show resolved Hide resolved

jenniexie reviewed Aug 27, 2019

View reviewed changes

src/test-apps/happy/tests/standalone/wdmNext/weave_wdm_next_test_base.py Outdated Show resolved Hide resolved

jenniexie reviewed Aug 27, 2019

View reviewed changes

src/test-apps/happy/test-templates/WeaveWdmNextOptions.py Show resolved Hide resolved

allenshihgoog force-pushed the feature/wdmNextRefactor branch from cb5d127 to cff7dc9 Compare September 3, 2019 17:46

Allen Shih and others added 9 commits November 6, 2019 15:32

wdmNext automation refactor improvements

e283026

fix absolute path of json param file

8c3d10c

experiment with grouping testcases and using dynamic testcase generation

a233307

push Makefile.in changes

40355e0

remove test grouping as it is slower without plaid in presubmit

2d59794

rebase from master

f1b8750

apply refactoring to wdm service tests

772a53e

fix service test tag name

4af608e

fix test_tag and add comment

ea7b7bc

allenshihgoog force-pushed the feature/wdmNextRefactor branch from cff7dc9 to ea7b7bc Compare November 7, 2019 00:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/wdm next refactor #341

Feature/wdm next refactor #341

allenshihgoog commented Aug 20, 2019

googlebot commented Aug 20, 2019

googlebot commented Aug 20, 2019

gerickson commented Aug 23, 2019

gerickson left a comment

codecov-io commented Aug 27, 2019 •

edited

Loading

allenshihgoog commented Aug 27, 2019

googlebot commented Aug 27, 2019

googlebot commented Aug 27, 2019

robszewczyk commented Aug 29, 2019

allenshihgoog commented Sep 17, 2019

Feature/wdm next refactor #341

Are you sure you want to change the base?

Feature/wdm next refactor #341

Conversation

allenshihgoog commented Aug 20, 2019

googlebot commented Aug 20, 2019

What to do if you already signed the CLA

Individual signers

Corporate signers

googlebot commented Aug 20, 2019

What to do if you already signed the CLA

Individual signers

Corporate signers

gerickson commented Aug 23, 2019

gerickson left a comment

Choose a reason for hiding this comment

codecov-io commented Aug 27, 2019 • edited Loading

Codecov Report

allenshihgoog commented Aug 27, 2019

googlebot commented Aug 27, 2019

googlebot commented Aug 27, 2019

robszewczyk commented Aug 29, 2019

allenshihgoog commented Sep 17, 2019

codecov-io commented Aug 27, 2019 •

edited

Loading