Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for console apps #1781

Merged
merged 26 commits into from
Jun 5, 2024
Merged

Add support for console apps #1781

merged 26 commits into from
Jun 5, 2024

Conversation

freakboy3742
Copy link
Member

@freakboy3742 freakboy3742 commented May 7, 2024

Adds support for packaging console apps with Briefcase.

In order to support console apps, it is also necessary to introduce a .pkg app packaging format for macOS. This generates an installer that is able to perform the post-installation step of adding a symlink to /usr/local/bin. The PKG format is the default and only packaging format allowed for macOS console apps.

This PR also renames the macOS app packaging format to zip - this is for consistency with Windows, to match the actual output format, and to avoid confusion about "packaging an app as an app that is a zip" descriptions.

To facilitate easier refactoring of the PKG format, the handling of signing identities on macOS has been modified to introduce a single SigningIdentity object that incorporates the id, identity name and team ID into a single object.

To enable generation of product GUIDs on Windows, a new dns_uuid5 template filter has been added. This will convert a domain name like feature.appname.example.com into a GUID using the UUID5 algorithm in the DNS namespace.

When running console apps (with dev or run), any output streaming is disabled. This is so that the user gets raw, unbuffered stdout - and especially stdin - access. However, when running a console app in test mode, streaming is enabled, as we need to be able to process test output looking for the sentinel.

This branch also adds a BRIEFCASE_DEBUG environment variable to the runtime environment if running in debug (i.e., -vv mode. One of the major changes to the stub binaries is that they no longer generate the "preamble" console output (detailing the PYTHONPATH etc) unless BRIEFCASE_DEBUG is enabled. This allows us to generate console apps that only generate output that the user actually requests, while retaining the ability to get deep debugging information if necessary.

Lastly, a new bootstrap (and a testing bootstrap) for console apps has been added.

This code can be tested with the macOS Xcode backend, the windows VisualStudio backend, the linux System backend the Flatpak backend. It requires template modifications to generate console-compatible apps. To test this branch, add:

template_branch = "console-app"
console_app = True

to your pyproject.toml, and run briefcase with:

$ briefcase run macOS Xcode
$ briefcase package macOS Xcode

The macOS App template and Windows App template have been partially updated. The logic to generate and handle 2 stub binaries has been added; however, those PRs cannot be completed until the macOS Xcode and Windows VisualStudio template changes have been merged. Inadvertently, this also fixes #1729; by moving the process of renaming the stub from the create step to the build step, we're able to clearly identify if the a build has been performed.

The landing strategy for this:

  • Land Add a UUID cookiecutter extension. #1850, adding the UUID cookiecutter template.
  • Do a preliminary review of this Briefcase branch
  • Test and land the non-app templates using this branch
  • Use this branch to manually generate stubs for the two app templates
  • Test and land the remaining templates
  • Do a final review and land this PR.

As an aside: This PR increases the need to address #993, as the app repos have just doubled the number of binaries that are stored. See #1849 for the in-progress work to extract the stub binaries.


Update: 4 Jun

The Xcode template updates have been landed, so briefcase run macOS Xcode and briefcase run Windows VisualStudio will work without the template_branch setting.

The app templates have been updated with new stub binaries, so briefcase run will work as long as the template_branch = "console-app" setting exists.

PR Checklist:

  • All new features have been tested
  • All new features have been documented
  • I have read the CONTRIBUTING.md file
  • I will abide by the code of conduct

@freakboy3742 freakboy3742 marked this pull request as ready for review May 11, 2024 03:51
@freakboy3742 freakboy3742 requested a review from mhsmith May 11, 2024 03:53

try:
icon_filename = self.base_path / f"{app.installer_icon}.icns"
# Console apps are installed in /Library/Formal Name, and include the

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't console apps also be installed in ~/Library/?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ~/Library installation location is an option if you enable user-specific installs. This implementation doesn't support creating user-specific installers, because there's no reliable user-space analog of /usr/local/bin

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

couldn't you soft link /usr/local/bin/app to the ~/Library/app executable in the postinstall the same way it does now? i guess come to think of it couldn't you just soft link straight to /Applications/app?

just asking because it seems like most of my installed apps are working with ~/Library via Application Support/ etc. instead of the system wide /Library and apple is only getting more and more strict about what is allowed to ever touch system wide stuff.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That approach won't work if there's more than one user.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Anthropic statement of work says we would install to /Applications. That seems to make more sense, because regardless of whether it's a console app or not, it is still an application and not a library.

If this changes, the postinstall script will need to be updated to match.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I picked the /Library location because macOS will surface any .app that exist in /Applications or /Users/xxx/Applications in the Launchpad, and while you would be able to launch a command line app, it wouldn't display a terminal as a result of being launched, so it would look like a no-op.

Putting console apps in /Library seemed broadly consistent with how tools like Xcode handle console-only tooling. It's a location that is clearly a "system" location; but doesn't result in command line apps surfacing in GUI representations.

It's also worth noting that the install location varies depending on the type of app. A GUI app will install into /Applications; a console app installs into /Library.

Copy link
Member

@mhsmith mhsmith Jun 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this also applies to subdirectories of /Applications – for example, the official Python installer creates a directory /Applications/Python 3.x containing IDLE.app and Python Launcher.app, both of which appear in the Launchpad.

Copy link
Member

@mhsmith mhsmith Jun 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, that makes sense, but since it's a change to the spec, let's remember to mention it to Anthropic next time we send them an update.

@michelcrypt4d4mus
Copy link

michelcrypt4d4mus commented May 21, 2024

i think you may just need to change the line

(installer_path / "resources").mkdir(exist_ok=True)

to add the parents=True option?

edit: changing that line fixed my issue.

(installer_path / "resources").mkdir(exist_ok=True, parents=True)

@freakboy3742
Copy link
Member Author

If you generated the project before you added the template_branch setting, then that will be the cause of the problem. The template is only used by the create command; once the template has been rolled out, the template isn't used again.

The difference between macOS Xcode and macOS app is that the Xcode template contains the full Xcode definition for building the stub app, and the app template contains a pre-compiled binary. Most projects can just use the pre-compiled binary, but obviously you need to generate the pre-compiled binary from somewhere - the Xcode project is also used to generate that binary.

@michelcrypt4d4mus
Copy link

michelcrypt4d4mus commented May 21, 2024

  1. what's the best practice for recreating my app with a different template? Right now I'm looking at moving pyproject.toml out of the way temporarily, running briefcase create, and then moving it back... but that seems a bit fraught.
  2. re: macOS Xcode and macOS app is the right way to think about it that they are basically the same but macOS Xcode leaves behind more artifacts in the form of the full xcodeproj etc?

@michelcrypt4d4mus
Copy link

nevermind - i sorted this out with briefcase create macOS Xcode. might want to update the PR description to add that step if other people might be testing this.

@michelcrypt4d4mus
Copy link

is there any mechanism for injecting custom build options into project.pbxproj? for instance i have customized the template with INFOPLIST_KEY_LSBackgroundOnly = YES; in the buildSettings:

0D354FDD2551BFBD119178D1 /* Debug */ = {
			isa = XCBuildConfiguration;
			buildSettings = {
				ARCHS = "$(ARCHS_STANDARD)";
				ASSETCATALOG_COMPILER_APPICON_NAME = "App Server";
				ASSETCATALOG_COMPILER_GLOBAL_ACCENT_COLOR_NAME = AccentColor;
				CLANG_CXX_LANGUAGE_STANDARD = "gnu++0x";
				CLANG_CXX_LIBRARY = "libc++";
				CODE_SIGN_ENTITLEMENTS = AppServer/app.entitlements;
				CODE_SIGN_IDENTITY = "-";
				CODE_SIGN_STYLE = Automatic;
				COMBINE_HIDPI_IMAGES = YES;
				ENABLE_HARDENED_RUNTIME = YES;
				GCC_C_LANGUAGE_STANDARD = gnu99;
				INFOPLIST_FILE = AppServer/Info.plist;
				INFOPLIST_KEY_LSBackgroundOnly = YES;
				LD_RUNPATH_SEARCH_PATHS = (
					"$(inherited)",
					"@executable_path/../Frameworks",
				);
				MACOSX_DEPLOYMENT_TARGET = 11.0;
				PRODUCT_BUNDLE_IDENTIFIER = com.appserver.app;
				PROVISIONING_PROFILE_SPECIFIER = "";
				STRIP_INSTALLED_PRODUCT = NO;
			};
			name = Debug;
		};

If I could somehow specify that in the briefcase config so that briefcase create didn't overwrite it that would be awesome.

@freakboy3742
Copy link
Member Author

nevermind - i sorted this out with briefcase create macOS Xcode. might want to update the PR description to add that step if other people might be testing this.

The PR description does say this:
Screenshot 2024-05-22 at 9 16 00 AM

is there any mechanism for injecting custom build options into project.pbxproj? for instance i have customized the template with INFOPLIST_KEY_LSBackgroundOnly = YES; in the buildSettings:

There's no mechanism to directly manipulate the project.pbxproj file; however, you can inject values into the Info.plist using the info key in your configuration. In your case, it sounds like adding info."LSBackgroundOnly" = true would do what you want.

@mhsmith
Copy link
Member

mhsmith commented May 29, 2024

  • Do a preliminary review of this Briefcase branch
  • Test and land the non-app templates (except for Windows VisualStudio) using this branch
  • Use this branch to manually generate stubs for the two app templates

Why not VisualStudio in step 2?

@freakboy3742
Copy link
Member Author

Why not VisualStudio in step 2?

It's a chicken and egg thing. The VisualStudio template requires the UUID change in this PR, so we can't land the template change without also landing this Briefcase branch. The same problem doesn't exist for the Xcode template because that template doesn't depend on any new Briefcase features.

If it would help, I could extract the UUID piece into a standalone PR; it's nicely self contained, and would then allow running the Visual Studio template against a mainline Briefcase.

docs/reference/platforms/macOS/app.rst Outdated Show resolved Hide resolved
docs/reference/platforms/macOS/app.rst Outdated Show resolved Hide resolved
docs/reference/platforms/macOS/app.rst Outdated Show resolved Hide resolved
docs/reference/platforms/macOS/xcode.rst Show resolved Hide resolved
docs/reference/configuration.rst Outdated Show resolved Hide resolved
@freakboy3742
Copy link
Member Author

If it would help, I could extract the UUID piece into a standalone PR; it's nicely self contained, and would then allow running the Visual Studio template against a mainline Briefcase.

I've opened #1850 to include just the UUID piece; I've modified the landing instructions to reflect the extra step of landing that PR.

Comment on lines +915 to +918
def package_pkg(
self,
app: AppConfig,
notarize_app: bool,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This argument is never used, and the .pkg is never notarized.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True - that's a leftover of a feature that I started looking at, then backed out of.

It is possible to sign and notarise installers; however, the signing process requires a separate signing identity to the app distribution signing identity.

This is a solvable problem - one that we should open as a feature ticket as soon as this PR lands - but it's a little fiddly, and AFAICT, it's not needed to use the installer in practice

My inclination is to leave the argument in, even though it isn't used, because it should be used; but I won't argue too hard if you think it should be removed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, let's keep it but leave a comment in the code explaining the situation.

To test this, we agreed that you'll sign a pkg and send it to me to make sure I can install it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had the sudden realization that I have a son with an M1 laptop (🤦), so I’ve just tested the installer on his machine. It works, but it triggers the “unknown developer” handling. It’s installable, but you’ve got to do the “explicit allow” dance through the Settings panel. So, it would appear that notarisation is a necessary step. Any preferences on rolling that change into this PR, vs a follow up?

Copy link
Member

@mhsmith mhsmith Jun 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've finished testing on all three platforms and found no other problems, so let's start merging now and do it as a follow-up. I've approved the Xcode and VisualStudio PRs.

Copy link

@michelcrypt4d4mus michelcrypt4d4mus Jun 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, it would appear that notarisation is a necessary step.

FWIW technically i don't think this is called "notarization"... that's a whole other process with a whole other developer certificate that you do before submitting to the app store. to get the installer to work without "unknown developer" issues (or at least none beyond right click -> install anyway) you can just run pkgbuild with the --sign "Developer ID Installer: Nasir Jones (XYZXYZXYZ)" (note that the Developer ID Installer is indeed a different certificate from the Apple Development certificate, as @freakboy3742 said).

my understanding - which is not 100% but i did manage to get a briefcase app into an installer successfully - is that there may be certificate types that can be used for both packaging and notarizing but there are definitely certificate types that will only do one or the other.

Screenshot 2024-06-04 at 1 08 27 AM

edit notarizing is actually a step done by apple - you submit your installer to them and they (somehow) notarize the installer to prep for submission to the app store. my understanding is that this is an optional but encouraged step.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok - technically signing is a required step; Notarization is an extra step that you might as well do once you're signing (especially given that we've got notarization ready to go).

However, the key detail is that you can't use the "Developer ID Application" certificate to sign (and thus notarize) the pkg archive, so additional certificate/identity handling is required.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this issue has some more of the details on the difference between signing and notarizing

Copy link

@michelcrypt4d4mus michelcrypt4d4mus Jun 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the key detail is that you can't use the "Developer ID Application" certificate to sign

correct

Notarization is an extra step that you might as well do once you're signing

it could very well be but tbh i'm not sure... as far as i can tell (and i could be wrong) to get an app notarized requires sending it to apple and getting something back from them. i don't know if there are any kind of limits on how many packages an app developer can get notarized but i'd be a bit wary of just automatically submitting to apple for notarization every time the package is built (at least not without fully understanding the apple notarization process).

Copy link
Member Author

@freakboy3742 freakboy3742 Jun 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it involves sending code to Apple and getting a response back. Briefcase already manages this process.

And yes, there is a rate limit. You don't want to put notarisation in a daily/every commit CI process, but there's no issue with having lots of apps, as long as you don't try to notarise them all on the same day.

@mhsmith
Copy link
Member

mhsmith commented Jun 4, 2024

With the macOS app template, briefcase run worked fine, but briefcase package gave this error:

[consoletest] Building PKG...
Installing license... errored

Log saved to /Users/msmith/git/beeware/apps/consoletest/logs/briefcase.2024_06_04-21_50_50.package.log

Traceback (most recent call last):
  File "/Users/msmith/.venv/bw-38/bin/briefcase", line 8, in <module>
    sys.exit(main())
  File "/Users/msmith/git/beeware/briefcase/src/briefcase/__main__.py", line 29, in main
    command(**options)
  File "/Users/msmith/git/beeware/briefcase/src/briefcase/commands/package.py", line 151, in __call__
    state = self._package_app(
  File "/Users/msmith/git/beeware/briefcase/src/briefcase/commands/package.py", line 99, in _package_app
    state = self.package_app(app, **full_options(state, options))
  File "/Users/msmith/git/beeware/briefcase/src/briefcase/platforms/macOS/__init__.py", line 878, in package_app
    self.package_pkg(
  File "/Users/msmith/git/beeware/briefcase/src/briefcase/platforms/macOS/__init__.py", line 931, in package_pkg
    (installer_path / "resources").mkdir(exist_ok=True)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pathlib.py", line 1288, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/resources'
(bw-38) msmith@mbp consoletest % ll /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/
total 16
drwxr-xr-x  3 msmith  staff   96 Jun  4 21:50 Console test.app
-rw-r--r--  1 msmith  staff  334 Jun  4 21:49 Entitlements.plist
drwxr-xr-x  2 msmith  staff   64 Jun  4 21:50 app_packages.x86_64
-rw-r--r--  1 msmith  staff  563 Jun  4 21:49 briefcase.toml
drwxr-xr-x  6 msmith  staff  192 Jun  4 21:50 support

I added installer_path.mkdir(exist_ok=True), but then I got this:

[consoletest] Building PKG...
Installing license... done
Copying app into products folder... done
Writing component manifest... done

>>> Running Command:
>>>     pkgbuild --root /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/root --component-plist /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/components.plist --install-location '/Library/Console test' --scripts /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/scripts /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/packages/consoletest.pkg
>>> Working Directory:
>>>     /Users/msmith/git/beeware/apps/consoletest
pkgbuild: error: Cannot write package to "/Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/packages/consoletest.pkg". (The file “scripts” couldn’t be opened.)
pkgbuild: Reading components from /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/components.plist
pkgbuild: Adding component at Console test.app
>>> Return code: 1
Building app package... done

>>> Running Command:
>>>     productbuild --distribution /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/Distribution.xml --package-path /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/packages --resources /Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/resources '/Users/msmith/git/beeware/apps/consoletest/dist/Console test-0.0.1.pkg'
>>> Working Directory:
>>>     /Users/msmith/git/beeware/apps/consoletest
productbuild: error: Specified distribution "/Users/msmith/git/beeware/apps/consoletest/build/consoletest/macos/app/installer/Distribution.xml" not found.
>>> Return code: 1
Building Console test-0.0.1.pkg... done

[consoletest] Packaged dist/Console test-0.0.1.pkg
(bw-38) msmith@mbp consoletest % echo $?
0

There are at least two problems here:

  • None of the things in the installer directory of the Xcode template are present in the app template, including resources/welcome.html, scripts/postinstall, and Distribution.xml.
  • Two consecutive commands failed, but Briefcase continued running and returned a successful exit code.

@mhsmith
Copy link
Member

mhsmith commented Jun 4, 2024

With the Windows app template, briefcase run and briefcase package worked fine, but after running the installer:

  • It created a Start menu shortcut to "C:\Program Files\Jane Developer\Console test\Console test.exe", but the file is actually called consoletest.exe. Unlike on macOS, this shortcut is not useless because it would open a terminal window, so it's probably worth fixing it rather than removing it.

  • The PATH is not updated, because the .wxs file updates aren't present in the app template.

@freakboy3742
Copy link
Member Author

freakboy3742 commented Jun 4, 2024

  • None of the things in the installer directory of the Xcode template are present in the app template, including resources/welcome.html, scripts/postinstall, and Distribution.xml.

Apologies - I had those modifications locally, but managed to not commit them. They're in the template now.

  • Two consecutive commands failed, but Briefcase continued running and returned a successful exit code.

Looks like I forgot to add check=True to the two pkgbuild and productbuild command invocations. Fixed.

With the Windows app template, briefcase run and briefcase package worked fine, but after running the installer:

  • It created a Start menu shortcut to "C:\Program Files\Jane Developer\Console test\Console test.exe", but the file is actually called consoletest.exe. Unlike on macOS, this shortcut is not useless because it would open a terminal window, so it's probably worth fixing it rather than removing it.

Agreed - I've added this to the app template, and opened beeware/briefcase-windows-VisualStudio-template#32 to add the same change to the VS project.

  • The PATH is not updated, because the .wxs file updates aren't present in the app template.

Fixed.

@mhsmith mhsmith merged commit 8e9a8f6 into beeware:main Jun 5, 2024
44 checks passed
@freakboy3742 freakboy3742 deleted the cmdline-app branch June 5, 2024 22:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants