Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(loader): update shade plugin for spark loader #566

Merged
merged 4 commits into from
Feb 24, 2024
Merged

fix(loader): update shade plugin for spark loader #566

merged 4 commits into from
Feb 24, 2024

Conversation

liuxiaocs7
Copy link
Member

@liuxiaocs7 liuxiaocs7 commented Jan 6, 2024 •
edited
Loading

Purpose of the PR

Main Changes

Verifying these changes

  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Already covered by existing tests, such as (please modify tests here).
  • Need tests and can be verified as follows:
    • xxx

Does this PR potentially affect the following parts?

  • Nope
  • Dependencies (add/update license info)
  • Modify configurations
  • The public API
  • Other affects (typed here)

Documentation Status

  • Doc - TODO
  • Doc - Done
  • Doc - No Need

@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working java Pull requests that update Java code loader hugegraph-loader labels Jan 6, 2024
@github-actions github-actions bot added the tools hugegraph-tools label Jan 6, 2024
@liuxiaocs7
Copy link
Member Author

liuxiaocs7 commented Jan 6, 2024 •
edited
Loading

exec

./bin/hugegraph-spark-loader.sh --master local[*] --name spark-hugegraph-loader --file ./example/spark/struct.json --host 192.168.110.214 --port 8080 --graph hugegraph

load data successfully.

@codecov Codecov
Copy link

codecov bot commented Jan 6, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (b066b80) 62.49% compared to head (d144216) 62.55%.
Report is 19 commits behind head on master.

Additional details and impacted files 
@@             Coverage Diff              @@
##             master     #566      +/-   ##
============================================
+ Coverage     62.49%   62.55%   +0.05%     
- Complexity     1903     1923      +20     
============================================
  Files           262      267       +5     
  Lines          9541     9677     +136     
  Branches        886      888       +2     
============================================
+ Hits           5963     6053      +90     
- Misses         3190     3235      +45     
- Partials        388      389       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Jan 6, 2024
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Jan 6, 2024
imbajin
imbajin reviewed Jan 8, 2024
View reviewed changes
Copy link
Member

@imbajin imbajin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

currently in the release-1.2.0, we can't load data well in spark-connector?

hugegraph-loader/assembly/static/bin/hugegraph-spark-loader.sh Show resolved Hide resolved
hugegraph-tools/pom.xml
@@ -115,6 +115,10 @@
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<artifactId>guava</artifactId>
<groupId>com.google.guava</groupId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where do we use it? (in spark-connector?)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

com.google.j2objc:j2objc-annotations conflict occurred after modifying the guava version to 30.0-jre

hugegraph-dist/scripts/dependency/known-dependencies.txt
failureaccess-1.0.1.jar
findbugs-annotations-1.3.9-1.jar
flatbuffers-1.2.0-3f79e055.jar
flatbuffers-java-1.9.0.jar
groovy-all-2.4.21.jar
gson-2.8.9.jar
guava-32.0.1-jre.jar
guava-30.0-jre.jar
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could give more conflicts context (maybe paste pic in the comments)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no guava conflict, but the version in hugegraph-common is 30.0-jre. The guava package does not seem to be used in the toolchain. Can it be inherited directly?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no guava conflict, but the version in hugegraph-common is 30.0-jre. The guava package does not seem to be used in the toolchain. Can it be inherited directly?

we do need to use common to manage guava version rather than define it separately (if common use 30.0, just keep up with it)

@liuxiaocs7
Copy link
Member Author

currently in the release-1.2.0, we can't load data well in spark-connector?

no, in loader/HugeGraphSparkLoader

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Feb 20, 2024
@imbajin
Copy link
Member

imbajin commented Feb 20, 2024

Note: we need basic test & more doc for the spark-loader & spark-connector (users may feel confused about them now)

@javeme javeme merged commit 26d6b31 into apache:master Feb 24, 2024
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@javeme javeme javeme approved these changes

@imbajin imbajin imbajin approved these changes

Assignees
No one assigned
Labels
bug Something isn't working java Pull requests that update Java code lgtm This PR has been approved by a maintainer loader hugegraph-loader size:L This PR changes 100-499 lines, ignoring generated files. tools hugegraph-tools
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Improve] Update loader package shade plugin
3 participants
@liuxiaocs7 @imbajin @javeme