Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating backend ImGui/ImPlot/ImNodes to latest versions #2275

Merged
merged 82 commits into from
Sep 17, 2024

Conversation

SamuMazzi
Copy link
Contributor

@SamuMazzi SamuMazzi commented Feb 7, 2024


name: Updating backend ImGui/ImPlot/ImNodes
about: Updated backends
title: Updating backend ImGui/ImPlot/ImNodes
assignees: ''


After a loooooot of work and time, we've finally finished the PR. A huge thank you to @v-ein for all the help and support.

Description

This PR aims to update the ImGui/ImPlot/ImNodes backends of DPG to way more recent versions.

Currently, DPG relies on really old version of these packages (~2/3 years old), now this is the situation before/after the PR:

  • ImGui: 1.83 (May 2021) -> 1.91.0 (July 2024)
  • ImPlot: 0.11 (May 2021) -> 0.17 (January 2024)
  • ImNodes: 0.5 (April 2022) -> updated to the latest available (March 2024)

Those versions are the latest available at the moment of merging the PR
and they are supported by DPG but this doesn't mean that we implemented
every single feature from those libraries (many are still missing like multi-selection,
several flags in different components, new API for shortcuts, etc...).

Below will be listed a summary of the major changes and new features. We tried to be as much backward compatible as possible. And hopefully you may not need to change anything in your current code.

Issues solved

Changes

  • Now there can be 3 x-axes other than the currently supported 3 y-axes
  • New StackToolWindow
  • New query API for plots. Now multiple queries can be created and managed. Check the demo and the new parameters in add_plot to have more info
    • Note: From now on, the query callback receives coordinates as ((x1, y1, x2, y2), (x1, y1, ...)). This differs from the old behavior where the callback was getting (x1, x2, y1, y2), i.e. first the X range, then the Y range. The new format describes a sequence of rectangles in the plot's coordinate space.
  • Now label parameter in add_separator will put a label inside the separator
  • ctrl + tab navigation across windows independent from keyboard_navigation parameter
  • Add Presentation section to MetricsTool like in the original ImGui
  • enable parameter in add_group will apply mvStyleVar_DisabledAlpha to all its children

Here is a list of all the new and deprecated arguments and functions.

Note: all deprecated functions and arguments are still accepted by DearPyGui API. Some of them may still work, some are ignored, but none should crash your application.

New functions

  • add_axis_tag
  • add_bar_group_series
  • add_inf_line_series
  • add_digital_series
  • get_plot_query_rects
  • set_axis_limits_constraints
  • reset_axis_limits_constraints
  • set_axis_zoom_constraints
  • reset_axis_zoom_constraints

New arguments

Function Arguments
add_2d_histogram_series
col_major
add_button
repeat
add_child_window






always_auto_resize
always_use_window_padding
auto_resize_x
auto_resize_y
frame_style
resizable_x
resizable_y
add_colormap_scale


format
mirror
reverse_dir
add_combo
fit_width
add_custom_series
no_fit
add_drag_line



delayed
no_cursor
no_fit
no_inputs
add_drag_point





clamped
delayed
no_cursor
no_fit
no_inputs
offset
add_group
enabled
add_heat_series
col_major
add_histogram_series

cumulative
horizontal
add_input_text





always_overwrite
auto_select_all
ctrl_enter_for_new_line
escape_clears_all
no_horizontal_scroll
no_undo_redo
add_line_series




loop
no_clip
segments
shaded
skip_nan
add_pie_series
ignore_hidden
add_plot







max_query_rects
min_query_rects
no_frame
no_inputs
override_mod
query_color
zoom_mod
zoom_rate
add_plot_axis











auto_fit
foreground_grid
no_highlight
no_initial_fit
no_label
no_menus
no_side_switch
opposite
pan_stretch
range_fit
scale
tick_format
add_plot_legend




no_buttons
no_highlight_axis
no_highlight_item
no_menus
sort
add_scatter_series
no_clip
add_stair_series

pre_step
shaded
add_stem_series
horizontal
add_subplots
share_series
add_table_column

angled_header
no_header_label
add_text_point
offset
add_tree_node

span_full_width
span_text_width
add_window
unsaved_document
configure_app



anti_aliased_fill
anti_aliased_lines
anti_aliased_lines_use_tex
docking_shift_only

Deprecated functions

  • add_hline_series: use add_inf_line_series()
  • add_vline_series: use add_inf_line_series()
  • get_plot_query_area: use get_plot_query_rects()
  • is_plot_queried: use get_plot_query_rects()

Deprecated arguments

Function Argument Explanation
add_histogram_series cumlative Deprecated because of a typo: use cumulative
add_image_button frame_padding Not supported anymore by Dear ImGui; still works in DPG but will eventually be removed.
add_plot anti_aliased Not supported by ImPlot anymore. To enable/disable anti-aliasing, use dpg.configure_app() with the anti_aliasing parameters.
add_plot no_child Removed in ImPlot as child windows are no longer needed to capture scroll.
add_plot no_highlight Removed because not supported by ImPlot anymore. To control the highlighting of series use the same argument in add_plot_legend.
add_plot

query_button
query_mod
This refers to the old way of querying in ImPlot, now replaced with add_drag_rect().
add_plot_axis log_scale Use scale=dpg.mvPlotScale_Log10 instead.
add_plot_axis time Use scale=dpg.mvPlotScale_Time instead.
add_text_point

x_offset
y_offset
Use the offset argument instead.

Nerd info (for developers)

Why?

ImGui is currently sponsored by big tech companies like Adobe, Blizzard, Activision, etc. so we can expect a lot of improvements with the update (there are already ton of them) and we can expect a lot of people working on the really ugly/hard things like graphical backend libraries.

At my workplace we are about to ship a product based on DPG. Because of my job, I had to edit the backend and then I found out that what I edited was already modified in the new ImGui and then I also found out that the ImGui version we were relying on in DPG was really old.
We want to stick with this library but I honestly told them that I don't see the point of relying on an almost dead (okay, no, maintenance mode) library, so we decided to either upgrade it or drop it and replace it with something else.

I know that this is quite a huge change, and that several problems can arise but I also think that it can be definitely worth it. Also with this huge community all of these changes can be "easily" tested.

OpenGL

Currently, for the build, DPG relies on gl3w in the examples folder of ImGui. This doesn't sound like a really stable thing and indeed a couple of releases later, that's been removed, so I added gl3w directly in the thirdparty folder.
Also ImGui decided to make some heavy changes in order to be more cross-platform. One of these, which is also my biggest concern about all of this work, is about the buttons of the keyboard. These are now less than before, and they have been all mapped to predefined values in ImGui.
So if someone used them heavily now they could have some problem.

Major concerns

  • As I said earlier, now there are less keyboard buttons than before, I think that they'll add more in the future, but I don't know how to solve this problem.
  • In my workplace we have never used ImNodes so we haven't tested that a lot.
    There haven't been any huge updates during these years so there shouldn't be a lot (or any) problems. If someone's willing to help and take a look, they'd be welcome.

Notes

There are still tons of updates and new features that can be implemented (i.e. Shortcuts API, Multi select, and many others), then if you want to take a look on what happened check this for ImPlot and ImGui

In ~2022 ImGui moved all the docking stuff in a separate branch, so now we're relying on that branch. This is not a big difference because it's always rebased on the main branch on every update and should be merged in the master in the version 2.0

Everything was tested with Valgrind in order to be sure that there wasn't any kind of accidental memory leak (or at least not more than those who were already there).

My main goal here was to have a real feedback from the most expert users on this work.

I hope I was clear enough!
I'd like to improve this library because I think it's really great, so any kind of suggestion will be well accepted.

@bandit-masked
Copy link
Collaborator

What a massive PR! It would be very cool if it would be merged. Grazie mille!

Is there a website for this product by your company?

@Atlamillias
Copy link
Contributor

This is a great PR!

It looks like a lot of the breaking changes could be made to not be if desired via aliases on the Python side, which is nice.

@SamuMazzi
Copy link
Contributor Author

What a massive PR! It would be very cool if it would be merged. Grazie mille!

Is there a website for this product by your company?

Really thank you!
Here it is the link for the product web page, a public demo should be available before the end of the second quarter!

@SamuMazzi
Copy link
Contributor Author

This is a great PR!

It looks like a lot of the breaking changes could be made to not be if desired via aliases on the Python side, which is nice.

Thank you! What do you mean with aliases? How would you use them in this case?

@bandit-masked
Copy link
Collaborator

Could you reply here when the demo is up? Or you could post on #showcase on Discord.

@bandit-masked
Copy link
Collaborator

Also, if you are looking to promote your app when it's done, I'd recommend posting on Reddit. I have some experience with that, so I'd could give a few suggestions if you like.

@Atlamillias
Copy link
Contributor

Atlamillias commented Feb 9, 2024

This is a great PR!

It looks like a lot of the breaking changes could be made to not be if desired via aliases on the Python side, which is nice.

Thank you! What do you mean with aliases? How would you use them in this case?

"Alias" may not have been the best word.

For example; in the case of discontinuing add_area_series, implementing a function in dearpygui.dearpygui named add_area_series that wraps add_line_series to allow for proper deprecation instead of discontinuing it altogether. Keyword arguments that would be no longer necessary or simply unsupported with this merge could be handled similarly, since most high-level dearpygui functions accept variadic keyword arguments. The behavior of some of these may differ given the update, but at least it wouldn't break the API.

While I personally don't mind when updates are API-breaking (a benefit of me not needing to work on large projects, I suppose), DPG is supposed to be in "maintenence mode"; @hoffstadt and @Pcothren may not want to have a major release, which would likely be necessary given the API change.

@bzczb
Copy link

bzczb commented Feb 10, 2024

This branch seems to work well with an app that uses DearPyPixl, just needed to rename mvTag to mvPlotTag so all the machinery worked.

@SamuMazzi
Copy link
Contributor Author

Also, if you are looking to promote your app when it's done, I'd recommend posting on Reddit. I have some experience with that, so I'd could give a few suggestions if you like.

Thank you, I appreciate that! I'll tell you something when it'll be the moment and I'll post something in the #showcase for sure

@SamuMazzi
Copy link
Contributor Author

This branch seems to work well with an app that uses DearPyPixl, just needed to rename mvTag to mvPlotTag so all the machinery worked.

That's great, thanks! I really need feedback of someone testing it!

@SamuMazzi
Copy link
Contributor Author

While I personally don't mind when updates are API-breaking (a benefit of me not needing to work on large projects, I suppose), DPG is supposed to be in "maintenence mode"; @hoffstadt and @Pcothren may not want to have a major release, which would likely be necessary given the API change.

Oh okay, got it, but I think this is gonna create a "complete" new version so maybe there won't be the need of making 100% backward compatible, but if they'll say so I'm sure we'll find a way... there aren't so many breaking changes

@bandit-masked
Copy link
Collaborator

Also, if you are looking to promote your app when it's done, I'd recommend posting on Reddit. I have some experience with that, so I'd could give a few suggestions if you like.
Thank you, I appreciate that! I'll tell you something when it'll be the moment and I'll post something in the #showcase for sure

The main things to take into account are:

  1. Day of the week and time of posting. There are websites dedicated to establishing the perfect day and time for each subreddit. The quick and simple rule is to post between 8 and 9 a.m. US East Coast time so that it can gather as many possible upvotes in a day as the post becomes trending as people start working.

  2. Post a GIF or video without any text if possible (whether that's possible depends on the subreddit). If it's just a GIF or video, the GIF or video will be shown directly in the feed, which draws more attention and upvotes to it. You can post a comment on your own post with the backstory and links. Unfortunately, r/python, which is one of the biggest communities, doesn't allow for posting a video only. I'm not sure, but it used to pick up the image associated with a GitHub link as the thumbnail for your post. Better make sure to test this, e.g. on an unknown/your own subreddit, before posting to the big subreddits.

  3. If you get a few upvotes really quickly in the beginning, that will boost the visibility incredibly and will start a domino effect. Hence, make sure people know you are going to post and get them to upvote within the first 10 minutes.

  4. Besides the obvious subreddits such r/python, find other subreddits to show your application. People tend to show a relatively stronger response in smaller, yet active communities, especially if its a good match.

Good luck!

@Atlamillias
Copy link
Contributor

Atlamillias commented Feb 18, 2024

This branch seems to work well with an app that uses DearPyPixl, just needed to rename mvTag to mvPlotTag so all the machinery worked.

Thanks for the heads-up! I'll need to test this PR against my apps that use DearPyPixl as well, although I don't expect I'll find many problems. Docking's one, I suppose.

@v-ein
Copy link
Contributor

v-ein commented Feb 27, 2024

While I really appreciate all the work done and still being done behind this PR, and welcome any fixes that correct possible earlier mistakes in ImGui upgrade, can we keep unrelated commits in separate PRs? (maybe published later, once this PR gets merged).

I've noticed that commits continue to trickle into this PR, and some of them look unrelated to ImGui upgrade, which increases the work to review all this and might cause extra havoc if this PR needs any substantial rework.

Unfortunately I myself haven't had a chance to review the PR yet - my apologies on that, I'm quite bogged with other stuff. Hoping to get some progress this week...

@SamuMazzi
Copy link
Contributor Author

While I really appreciate all the work done and still being done behind this PR, and welcome any fixes that correct possible earlier mistakes in ImGui upgrade, can we keep unrelated commits in separate PRs? (maybe published later, once this PR gets merged).

Yeah sure! I think you refer to the "recursive" functionality (as far as I remember it should be the only "extra" thing that I added)... I was actually uncertain of pushing that commit, but I'll revert it asap

And obviously no problem for the review :)

@v-ein
Copy link
Contributor

v-ein commented Mar 10, 2024

Just a heads up - I'm reviewing this PR as time permits. It's hard to estimate progress - I think I'm still less than 20% into it. Since there's still a lot to review, I'm not making any comments yet - will be adding them later.

@ArneSchulzTUBS
Copy link

Thanks for the great work. How much effort would it be to directly use ImGui's docking branch so that we can keep the docking feature in DearPyGui ? If i understand correctly, their main and docking branch are regularly synced so there aren't too many differences in the core functionality.

@v-ein
Copy link
Contributor

v-ein commented Mar 14, 2024

+1 for keeping the docking feature. That's one of the things I was going to discuss.

@v-ein
Copy link
Contributor

v-ein commented Mar 27, 2024

While I'm still going through all the changes, I think it's time to start making some initial comments.

First of all, a disclaimer: the text below reflects my own opinion and not that of DPG devs (I'm not one of DPG authors). Also, some of my comments might seem a bit harsh - please bear with me, it's not because of any negative attitude, it's more like giving up on proper wording 🙂.

Here are some key things (other comments will be attached to code lines as usual in a PR).

  • I'm highly upset with the way backward compatibility is handled in this PR. In my opinion, we should do a better job on it, to minimize the hassle for authors of DPG-based projects - and for their users who update DPG and then find that their favorite application no longer works (I'm sure I've seen unhappy comments from such users on earlier breaking changes in DPG).

    IMHO, every breaking change must have a stone solid reason behind it, and sometimes I couldn't find such a reason. I might be making more comments on this in regard to particular changes.

  • Also about compatibility: I'm not sure how much effort it would be to keep the docking feature, but as a DPG user I would definitely want to retain it.

    I think DPG was originally built based on the docking branch of ImGui, but I might be wrong (well, docking started in 2018 and ImGui 1.83 is from 2021). In ImGui, docking periodically (and quite frequently) pulls changes from master, so I don't see a reason why we can't build it on docking now.

  • As soon as this PR gets approved, it's going to be merged into the master branch, and there won't be a second chance for a cleanup. This means all debugging/diagnostic thingies must be cleaned out ASAP, lest they slip through into master. This includes all mentions of <iostream> and std::cout.

    Yes, I've seen a note on this in the PR description; what I'm afraid of is that this PR will not be treated as a draft but rather as a "production PR", will get merged, and we'll see those in the production code. (BTW, I'm treating it as "prod" now because it's not clear how to handle comments, changes, etc. if we create yet another PR).

  • A couple of words on the doc, including descriptions for all the new arguments, functions, etc. DPG is often used by people who have little programming experience overall, and absolutely no Dear ImGui knowledge. With that, any Dear ImGui terms that are not reflected in the DPG, let alone names of ImGui internals, will cause a lot of questions if mentioned in the doc.

    I can make a number of comments like "Needs a rewrite - this is an ImGui term", but I'd suggest that we go ahead and revisit the changes in the doc now (mainly mvAppItem.cpp) without me finger-pointing all the ImGui thingies.

    • This applies to some arguments names too, which might be rather counter-intuitive for non-ImGui people. For these, I'll explicitly comment on them in the code. Personally I think that, while it's good to stick to original flag/function names (that come from ImGui/ImPlot/ImNodes), we don't have to sacrifice clarity for that, and can give something a different name if it makes sense in the context.

Making comments in the PR will take quite some time (maybe a couple of days), and I'm still halfway through the review, so get your popcorn ready! 🍿 😁

@v-ein v-ein mentioned this pull request Mar 28, 2024
@v-ein
Copy link
Contributor

v-ein commented Mar 28, 2024

One more thing I forgot to mention:

I'd like to ask a favor of whoever makes changes in future - in other PRs as well! - to avoid unnecessary re-formatting, renaming etc. because it pollutes git blame, often for no reason other than taste. git blame is a very useful tool when it comes to understanding why the code was written the way it was written. Nothing critical, just keep that in mind.

This is also what I actually mean when I type something like "the old code worked the same way, why change it?"

dearpygui/dearpygui.py Show resolved Hide resolved
src/dearpygui_parsers.h Outdated Show resolved Hide resolved
src/dearpygui_parsers.h Show resolved Hide resolved
src/dearpygui_commands.h Show resolved Hide resolved
@v-ein
Copy link
Contributor

v-ein commented Mar 28, 2024

A couple of words on set_decimal_point:

At the moment, it is broken for add_input_float, add_slider_float, add_drag_float (and their double counterparts). That is, it only seems to work on add_input_text with either decimal=True or scientific=True. See this comment in ImGui ticket #6719.

Do we want to add a feature to DPG in such a broken state? If so, we need to make a warning either in the release notes or in the doc (or both), and eventually update ImGui once again to pull up the fix for this.

src/dearpygui_commands.h Outdated Show resolved Hide resolved
src/dearpygui_commands.h Outdated Show resolved Hide resolved
src/mvGlobalHandlers.cpp Show resolved Hide resolved
src/mvGlobalHandlers.cpp Outdated Show resolved Hide resolved
src/mvGlobalHandlers.cpp Outdated Show resolved Hide resolved
src/mvBasicWidgets.cpp Outdated Show resolved Hide resolved
src/mvTables.cpp Outdated Show resolved Hide resolved
src/mvTables.cpp Outdated Show resolved Hide resolved
src/mvTables.cpp Show resolved Hide resolved
src/mvPlotting.cpp Outdated Show resolved Hide resolved
src/mvPlotting.cpp Outdated Show resolved Hide resolved
- remove useless `no_legend` parameter in `custom_series`
- remove `delete_rect` parameter in `plot`
- change name of parameter `invert_order` to `invert`
Specifically:

- remove build files
- restore old `README.md`
- add possibility to color query rect in plots
- effectively enable `query` parameter in plots
- fix deletion of query rects
- fix deprecated `add_vline_series` and `add_hline_series`
- remove `no_legend` parameter for subplots
- correct descriptions
- remove useless part with `IMGUI_DISABLE_OBSOLETE_FUNCTIONS`
- fix validation of BarGroupPlot paramters
Also, remove useless `no_label` parameter from `mvColorMapScale`
Also:
- remove `no_label` parameter in colormap scale
- fix `PlotStems` flags
- fix `share_series` parameter in Subplots
Also rename `resizable_x/y` left from previous change together with a couple of other parameters' name fixes
Specifically:
- Get anti-alias info from `get_app_configuration`
- Remove `offset` parameter from `digital_series`
- Remove/comment out leftovers
- Fix typo
- Improve docs
Also, replace deprecated parameters in `dpg.draw_rectangle`in demo
@SamuMazzi
Copy link
Contributor Author

@v-ein not the best choice of my life that rebase, but here weare, I did it! 😄

Copy link
Contributor

@v-ein v-ein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we're ready for merge! @hoffstadt

@v-ein
Copy link
Contributor

v-ein commented Aug 1, 2024

In case somebody is interested, here are the test scripts that I used to check all those new functions and arguments:

https://github.com/v-ein/dpg-tests-2275

@hoffstadt
Copy link
Owner

Alright guys... starting my part of this now.

@hoffstadt hoffstadt added the priority: high high priority label Sep 16, 2024
Copy link
Owner

@hoffstadt hoffstadt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Guys... great work. This looks great. I will approve it.

Before merging, I'm going to put the current master into another branch and update the readthedocs account to support multiple versions (dpg 1.0 & 2.0).

I can't thank you guys enough for all the work. I apologize for the state of the code you guys had to suffer through. It was in the middle of a refactor that I lost all motivation to finish.

@hoffstadt hoffstadt merged commit 2b4ccce into hoffstadt:master Sep 17, 2024
4 checks passed
@hoffstadt
Copy link
Owner

Currently on pypi as 2.0.0b1:

pip install dearpygui==2.0.0b1

@v-ein
Copy link
Contributor

v-ein commented Oct 3, 2024

I know it's a bit late in the process (since the PR has been merged), but can we add a line to the description @SamuMazzi?

It turns out that the order of coordinates in the new query callback is different from what it was in the old version. Let's add this line (as a sub-item probably) to the "New query API for plots" line:


Note: From now on, the query callback receives coordinates as ((x1, y1, x2, y2), (x1, y1, ...)). This differs from the old behavior where the callback was getting (x1, x2, y1, y2), i.e. first the X range, then the Y range. The new format describes a sequence of rectangles in the plot's coordinate space.

@SamuMazzi
Copy link
Contributor Author

Did it! Thanks for telling me that :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: high high priority
Projects
None yet
8 participants