M1 mac compatibility #5511

style95 · 2024-09-17T07:20:16Z

This is a very first step to make OW compatible with M1 Mac.

Description

There could be many ways but I chose to minimize changes. I decided to have another Dockerfile for the M1 Mac. It will be selectively used according to the OS and its architecture.

I could build and deploy OW on M1 Mac and run an action.
(Though I had to install rosetta because openwhisk-cli is also not being built for M1 mac.)

$ wsk action invoke noop -i -r
{}

Related issue and scope

I opened an issue to propose and discuss this change (Produce multi-arch docker images #5503)

My changes affect the following components

Types of changes

Bug fix (generally a non-breaking change which closes an issue).
Enhancement or new feature (adds new functionality).
Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

I signed an Apache CLA.
I reviewed the style guides and followed the recommendations (Travis CI will check :).
I added tests to cover my changes.
My changes require further changes to the documentation.
I updated the documentation where necessary.

style95 · 2024-09-17T07:23:56Z

ansible/group_vars/all

-  version: 3.4
+  image:
+    amd64: zookeeper:3.4
+    arm64: arm64v8/zookeeper:3.4


I added another zookeeper image for arm64

style95 · 2024-09-17T07:26:00Z

ansible/group_vars/all

@@ -464,7 +466,7 @@ zeroDowntimeDeployment:
  enabled: "{{ zerodowntime_deployment_switch | default(false) }}"

 etcd:
-  version: "{{ etcd_version | default('v3.4.33') }}"
+  version: "{{ etcd_version | default('3.5') }}"


etcd supports multiarch since 3.5 version.
https://hub.docker.com/r/bitnami/etcd/

I know this is stale now, but does etcd still require one to enable experimental features to run on arm?

Ignore that, I found the answer. Looks like arm64 has first-class support now as of 3.5. Previously it required one to use the ETCD_UNSUPPORTED_ARCH environment variable.

style95 · 2024-09-17T07:26:33Z

ansible/roles/invoker/tasks/deploy.yml

@@ -80,7 +80,7 @@
  when: environmentInformation.type == "docker-machine"

 - name: "determine docker root dir"
-  shell: echo -e "GET http:/v1.24/info HTTP/1.0\r\n" | nc -U /var/run/docker.sock | grep "{"


It doesn't work with the recent version of docker for mac.

style95 · 2024-09-17T07:27:02Z

ansible/roles/zookeeper/tasks/deploy.yml

+- name: set zookeeper image
+  set_fact:
+    zookeeper_image: "{{ zookeeper.image.arm64 }}"
+  when: ansible_facts['os_family'] == "Darwin" and ansible_facts['architecture'] == "arm64"


Selectively use the arm64 zookeeper image according to the os and architecture.

style95 · 2024-09-17T07:27:47Z

common/scala/Dockerfile.arm

+# limitations under the License.
+#
+# if you change version of openjsk, also update tools/github/setup.sh to download the corresponding jdk
+FROM arm64v8/eclipse-temurin:21.0.4_7-jdk-alpine


Only the base image is changed.

style95 · 2024-09-17T07:28:27Z

common/scala/build.gradle

@@ -24,6 +24,9 @@ plugins {
 }

 ext.dockerImageName = 'scala'
+if(System.getProperty("os.arch").toLowerCase(Locale.ENGLISH).startsWith("aarch")) {


If the architecture of OS is m1 Mac, it uses the Dockerfile with the .arm suffix.

style95 · 2024-09-17T07:31:41Z

core/scheduler/build.gradle

+protobuf {
+    protoc {
+        if (osdetector.os == "osx") {
+            artifact = 'com.google.protobuf:protoc:3.4.0:osx-x86_64'


IIRC, we cant' upgrade the protoc version for now because some libraries are dependent on this version.
And protoc-3.4.0 does not support arm architecture yet, so I chose this way.
This also requires rosetta installed.

grpc/grpc-java#7690 (comment)

style95 · 2024-09-17T07:34:58Z

tools/ow-utils/Dockerfile.arm

+# limitations under the License.
+#
+
+FROM arm64v8/eclipse-temurin:8u422-b05-jdk-noble


I only changed the base image from adoptopenjdk/openjdk8:jdk8u262-b10 to arm64v8/eclipse-temurin:8u422-b05-jdk-noble.

I used it as the current image is deprecated.

https://hub.docker.com/_/adoptopenjdk

style95 · 2024-09-17T08:41:18Z

One concern is we don't have any CI pipelines for the m1 Mac.
It seems the M1 runner is available for all plans. It would be great if we set up one pipeline.

actions/runner-images#9254

dgrove-oss · 2024-09-17T15:33:59Z

One concern is we don't have any CI pipelines for the m1 Mac. It seems the M1 runner is available for all plans. It would be great if we set up one pipeline.

actions/runner-images#9254

My suggestion would be that we go after enabling one of the CI pipelines for both x86_64 and arm in a separate PR.

dgrove-oss

This all makes sense to me. Thank you for getting started on this @style95!

I have not had a chance to try it on my Mac yet, but hope to carve out some time this afternoon.

dgrove-oss · 2024-09-17T21:33:42Z

Definitely an improvement; I was able to do distDocker and install successfully on my main laptop for the first time in several years.

style95

Let me try deploying this branch on a linux machine before merging to do a basic sanity check.

style95 · 2024-09-19T00:12:48Z

ansible/roles/etcd/tasks/deploy.yml

    state: started
    recreate: true
    restart_policy: "{{ docker.restart.policy }}"
    volumes: "{{volume_dir | default([])}}"
-    command: "/usr/local/bin/etcd \


Before merging, I will check how to enable these options with the Bitnami/etcd image.

I think this can be addressed in a subsequent PR.

style95 · 2024-09-19T10:57:05Z

I could build and deploy OW with this branch and successfully invoke an action with a Linux machine.

style95 · 2024-09-22T04:20:08Z

I am merging this as it has been open for a while.

style95 added 6 commits September 17, 2024 16:02

Apply x86 protoc for m1 mac

fd10ea3

Enable the Dockerfile suffix for m1 Mac

9dde236

Enable the Dockerfile suffix for ow-utils

2827fdc

Choose zookeeper image according to the os architecture

0d6598e

Use the bitnami/etcd image that supports arm

30bf2c5

Replace docker info API to cli command

cef3d29

style95 requested a review from dgrove-oss September 17, 2024 07:20

style95 commented Sep 17, 2024

View reviewed changes

style95 requested a review from rabbah September 17, 2024 07:36

dgrove-oss approved these changes Sep 17, 2024

View reviewed changes

style95 commented Sep 19, 2024

View reviewed changes

style95 merged commit daac765 into apache:master Sep 22, 2024
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

M1 mac compatibility #5511

M1 mac compatibility #5511

style95 commented Sep 17, 2024

style95 Sep 17, 2024

style95 Sep 17, 2024

shawnallen85 Sep 27, 2024

shawnallen85 Sep 27, 2024

style95 Sep 17, 2024

style95 Sep 17, 2024

style95 Sep 17, 2024

style95 Sep 17, 2024

style95 Sep 17, 2024

style95 Sep 17, 2024

style95 commented Sep 17, 2024 •

edited

Loading

dgrove-oss commented Sep 17, 2024

dgrove-oss left a comment

dgrove-oss commented Sep 17, 2024 •

edited

Loading

style95 left a comment

style95 Sep 19, 2024

style95 Sep 19, 2024

style95 commented Sep 19, 2024 •

edited

Loading

style95 commented Sep 22, 2024

M1 mac compatibility #5511

M1 mac compatibility #5511

Conversation

style95 commented Sep 17, 2024

Description

Related issue and scope

My changes affect the following components

Types of changes

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

style95 commented Sep 17, 2024 • edited Loading

dgrove-oss commented Sep 17, 2024

dgrove-oss left a comment

Choose a reason for hiding this comment

dgrove-oss commented Sep 17, 2024 • edited Loading

style95 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

style95 commented Sep 19, 2024 • edited Loading

style95 commented Sep 22, 2024

style95 commented Sep 17, 2024 •

edited

Loading

dgrove-oss commented Sep 17, 2024 •

edited

Loading

style95 commented Sep 19, 2024 •

edited

Loading