Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add script for converting imports in grpc python modules and remove 2to3 #1557

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Add script for converting imports in grpc python modules and remove 2to3 #1557

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f899968
Add script for converting imports in grpc python modules and remove 2to3
lithorus Oct 24, 2024
ad902f5
Remove debug prints
lithorus Oct 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ci/build_sphinx_docs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ python -m grpc_tools.protoc -I=proto/ --python_out=pycue/opencue/compiled_proto

# Fix imports to work in both Python 2 and 3. See
# <https://github.com/protocolbuffers/protobuf/issues/1491> for more info.
2to3 -wn -f import pycue/opencue/compiled_proto/*_pb2*.py
python ci/fix_compiled_proto.py pycue/opencue/compiled_proto

# Build the docs and treat warnings as errors
~/.local/bin/sphinx-build -W -b html -d docs/_build/doctrees docs docs/_build/html
26 changes: 26 additions & 0 deletions ci/fix_compiled_proto.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/usr/bin/env python

"""Script that makes the imports in the generated compiled_proto python files relative.

"""
import os
import re
import sys
import glob

PYTHON_SCRIPT_PATH = sys.argv[1]

if os.path.isdir(PYTHON_SCRIPT_PATH):
pattern = re.compile(r"^import \w+ as \w+_pb2")
for filepath in glob.glob(os.path.join(PYTHON_SCRIPT_PATH, "*_pb2*.py")):
filedata = []
with open(filepath) as f:
for line in f.readlines():
match = pattern.match(line)
if match is not None:
line = f"from . {line}"
filedata.append(line.strip("\n"))
with open(filepath, "w") as f:
f.write("\n".join(filedata))
else:
print("Argument is not a directory")
4 changes: 2 additions & 2 deletions ci/python_coverage_report.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ python -m pip install coverage pytest-xvfb
# Protos need to have their Python code generated in order for tests to pass.
python -m grpc_tools.protoc -I=proto/ --python_out=pycue/opencue/compiled_proto --grpc_python_out=pycue/opencue/compiled_proto proto/*.proto
python -m grpc_tools.protoc -I=proto/ --python_out=rqd/rqd/compiled_proto --grpc_python_out=rqd/rqd/compiled_proto proto/*.proto
2to3 -wn -f import pycue/opencue/compiled_proto/*_pb2*.py
2to3 -wn -f import rqd/rqd/compiled_proto/*_pb2*.py
python ci/fix_compiled_proto.py pycue/opencue/compiled_proto
python ci/fix_compiled_proto.py rqd/rqd/compiled_proto

# Run coverage for each component individually, but append it all into the same report.
python -m coverage run --source=pycue/opencue/,pycue/FileSequence/ --omit=pycue/opencue/compiled_proto/* pycue/tests/test_suite.py
Expand Down
4 changes: 2 additions & 2 deletions ci/run_python_lint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ python -m grpc_tools.protoc -I=proto/ --python_out=rqd/rqd/compiled_proto --grpc

# Fix imports to work in both Python 2 and 3. See
# <https://github.com/protocolbuffers/protobuf/issues/1491> for more info.
2to3 -wn -f import pycue/opencue/compiled_proto/*_pb2*.py
2to3 -wn -f import rqd/rqd/compiled_proto/*_pb2*.py
python ci/fix_compiled_proto.py pycue/opencue/compiled_proto
python ci/fix_compiled_proto.py rqd/rqd/compiled_proto

echo "Running lint for pycue/..."
cd pycue
Expand Down
16 changes: 8 additions & 8 deletions ci/run_python_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,14 @@ python -m grpc_tools.protoc -I=proto/ --python_out=rqd/rqd/compiled_proto --grpc

# Fix imports to work in both Python 2 and 3. See
# <https://github.com/protocolbuffers/protobuf/issues/1491> for more info.
2to3 -wn -f import pycue/opencue/compiled_proto/*_pb2*.py
2to3 -wn -f import rqd/rqd/compiled_proto/*_pb2*.py

python3 -m unittest discover -s pycue/tests -t pycue -p "*.py"
PYTHONPATH=pycue python3 -m unittest discover -s pyoutline/tests -t pyoutline -p "*.py"
PYTHONPATH=pycue python3 -m unittest discover -s cueadmin/tests -t cueadmin -p "*.py"
PYTHONPATH=pycue:pyoutline python3 -m unittest discover -s cuesubmit/tests -t cuesubmit -p "*.py"
python3 -m unittest discover -s rqd/tests -t rqd -p "*.py"
python ci/fix_compiled_proto.py pycue/opencue/compiled_proto
python ci/fix_compiled_proto.py rqd/rqd/compiled_proto

python -m unittest discover -s pycue/tests -t pycue -p "*.py"
PYTHONPATH=pycue python -m unittest discover -s pyoutline/tests -t pyoutline -p "*.py"
PYTHONPATH=pycue python -m unittest discover -s cueadmin/tests -t cueadmin -p "*.py"
PYTHONPATH=pycue:pyoutline python -m unittest discover -s cuesubmit/tests -t cuesubmit -p "*.py"
python -m unittest discover -s rqd/tests -t rqd -p "*.py"


# Xvfb no longer supports Python 2.
Expand Down
2 changes: 1 addition & 1 deletion connectors/prometheus_metrics/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ RUN python -m grpc_tools.protoc \

# Fix imports to work in both Python 2 and 3. See
# <https://github.com/protocolbuffers/protobuf/issues/1491> for more info.
RUN 2to3 -wn -f import pycue/opencue/compiled_proto/*_pb2*.py
RUN python3 ci/fix_compiled_proto.py pycue/opencue/compiled_proto

RUN cd pycue && python setup.py install

Expand Down
2 changes: 1 addition & 1 deletion cueadmin/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ RUN python3 -m grpc_tools.protoc \

# Fix imports to work in both Python 2 and 3. See
# <https://github.com/protocolbuffers/protobuf/issues/1491> for more info.
RUN 2to3 -wn -f import pycue/opencue/compiled_proto/*_pb2*.py
RUN python3 ci/fix_compiled_proto.py pycue/opencue/compiled_proto

COPY cueadmin/README.md ./cueadmin/
COPY cueadmin/setup.py ./cueadmin/
Expand Down
2 changes: 1 addition & 1 deletion cuegui/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ RUN python3.6 -m grpc_tools.protoc \

# Fix imports to work in both Python 2 and 3. See
# <https://github.com/protocolbuffers/protobuf/issues/1491> for more info.
RUN 2to3 -wn -f import pycue/opencue/compiled_proto/*_pb2*.py
RUN python3 ci/fix_compiled_proto.py pycue/opencue/compiled_proto

COPY cuegui/README.md ./cuegui/
COPY cuegui/setup.py ./cuegui/
Expand Down
2 changes: 1 addition & 1 deletion cuesubmit/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ RUN python3.6 -m grpc_tools.protoc \

# Fix imports to work in both Python 2 and 3. See
# <https://github.com/protocolbuffers/protobuf/issues/1491> for more info.
RUN 2to3 -wn -f import pycue/opencue/compiled_proto/*_pb2*.py
RUN python3 ci/fix_compiled_proto.py pycue/opencue/compiled_proto

COPY pyoutline/README.md ./pyoutline/
COPY pyoutline/setup.py ./pyoutline/
Expand Down
10 changes: 4 additions & 6 deletions proto/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,14 @@ To generate:

```sh
python -m grpc_tools.protoc -I=. --python_out=../rqd/rqd/compiled_proto --grpc_python_out=../rqd/rqd/compiled_proto ./*.proto
2to3 -wn -f import ../rqd/rqd/compiled_proto/*_pb2*.py
python ../ci/fix_compiled_proto.py ../rqd/rqd/compiled_proto
```

For Windows (Powershell):

```powershell
python -m grpc_tools.protoc --proto_path=. --python_out=../rqd/rqd/compiled_proto --grpc_python_out=../rqd/rqd/compiled_proto (ls *.proto).Name
cd ..\rqd\rqd\compiled_proto\
2to3 -wn -f import (ls *_pb2*.py).Name
python ../ci/fix_compiled_proto.py ../rqd/rqd/compiled_proto
```


Expand All @@ -32,15 +31,14 @@ To generate:

```sh
python -m grpc_tools.protoc -I=. --python_out=../pycue/opencue/compiled_proto --grpc_python_out=../pycue/opencue/compiled_proto ./*.proto
2to3 -wn -f import ../pycue/opencue/compiled_proto/*_pb2*.py
python ../ci/fix_compiled_proto.py ../pycue/opencue/compiled_proto
```

For Windows (Powershell):

```powershell
python -m grpc_tools.protoc --proto_path=. --python_out=../pycue/opencue/compiled_proto --grpc_python_out=../pycue/opencue/compiled_proto (ls *.proto).Name
cd ..\pycue\opencue\compiled_proto\
2to3 -wn -f import (ls *_pb2*.py).Name
python ../ci/fix_compiled_proto.py ../pycue/opencue/compiled_proto
```


2 changes: 1 addition & 1 deletion pycue/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ RUN python3 -m grpc_tools.protoc \

# Fix imports to work in both Python 2 and 3. See
# <https://github.com/protocolbuffers/protobuf/issues/1491> for more info.
RUN 2to3 -wn -f import pycue/opencue/compiled_proto/*_pb2*.py
RUN python ci/fix_compiled_proto.py pycue/opencue/compiled_proto

COPY VERSION.in VERSIO[N] ./
RUN test -e VERSION || echo "$(cat VERSION.in)" | tee VERSION
Expand Down
2 changes: 1 addition & 1 deletion pyoutline/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ RUN python3 -m grpc_tools.protoc \

# Fix imports to work in both Python 2 and 3. See
# <https://github.com/protocolbuffers/protobuf/issues/1491> for more info.
RUN 2to3 -wn -f import pycue/opencue/compiled_proto/*_pb2*.py
RUN python ci/fix_compiled_proto.py pycue/opencue/compiled_proto

COPY pyoutline/README.md ./pyoutline/
COPY pyoutline/setup.py ./pyoutline/
Expand Down
1 change: 0 additions & 1 deletion requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
2to3==1.0
enum34==1.1.6
future==1.0.0
grpcio==1.48.2;python_version<"3.7"
Expand Down
2 changes: 1 addition & 1 deletion rqd/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ RUN python3.9 -m grpc_tools.protoc \

# Fix imports to work in both Python 2 and 3. See
# <https://github.com/protocolbuffers/protobuf/issues/1491> for more info.
RUN 2to3 -wn -f import rqd/rqd/compiled_proto/*_pb2*.py
RUN python ci/fix_compiled_proto.py rqd/rqd/compiled_proto

COPY VERSION.in VERSIO[N] ./
RUN test -e VERSION || echo "$(cat VERSION.in)" | tee VERSION
Expand Down
2 changes: 1 addition & 1 deletion sandbox/install-client-sources.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ python -m grpc_tools.protoc -I=. \
--python_out=../pycue/opencue/compiled_proto \
--grpc_python_out=../pycue/opencue/compiled_proto ./*.proto
cd ..
2to3 -wn -f import pycue/opencue/compiled_proto/*_pb2*.py
python ../ci/fix_compiled_proto.py pycue/opencue/compiled_proto

# Install all client packages.
pip install pycue/ pyoutline/ cueadmin/ cuesubmit/ cuegui/
Loading