Documentation site

.readthedocs.yaml

@@ -0,0 +1,33 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+   os: ubuntu-22.04
+   tools:
+     python: "3.10"
+     # You can also specify other tool versions:
+     # nodejs: "16"
+     # rust: "1.55"
+     # golang: "1.17"
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+   configuration: docs/conf.py
+   fail_on_warning: true
+
+# If using Sphinx, optionally build your docs in additional formats such as PDF
+# formats:
+#    - pdf
+
+# Optionally declare the Python requirements required to build your docs
+python:
+   install:
+   - requirements: docs/requirements.txt
+   - requirements: requirements.txt
+   - method: pip
+     path: .

README.md

@@ -4,17 +4,20 @@
 
 [![Version](https://img.shields.io/pypi/v/deepbots?color=green)](https://pypi.org/project/deepbots/)
 [![Dev Version](https://img.shields.io/github/v/tag/aidudezzz/deepbots?include_prereleases&label=test-pypi&color=green)](https://test.pypi.org/project/deepbots/)
+[![Documentation Status](https://readthedocs.org/projects/deepbots/badge/?version=latest)](https://deepbots.readthedocs.io/en/latest/?badge=latest)
 [![Downloads](https://static.pepy.tech/personalized-badge/deepbots?period=total&units=international_system&left_color=grey&right_color=green&left_text=Downloads)](https://pepy.tech/project/deepbots)
 [![License](https://img.shields.io/github/license/aidudezzz/deepbots?color=green)](https://github.com/aidudezzz/deepbots/blob/dev/LICENSE)
 [![All Contributors](https://img.shields.io/badge/all_contributors-6-orange.svg?style=flat-square)](#contributors-)
 
 Deepbots is a simple framework which is used as "middleware" between the free
 and open-source [Cyberbotics' Webots](https://cyberbotics.com/) robot simulator
-and Reinforcement Learning algorithms. When it comes to Reinforcement Learning
-the [OpenAI gym](https://gym.openai.com/) environment has been established as
-the most used interface between the actual application and the RL algorithm.
-Deepbots is a framework which follows the OpenAI gym environment interface
-logic in order to be used by Webots applications.
+and Reinforcement Learning (RL) algorithms. When it comes to RL, 
+[gym](https://www.gymlibrary.dev/) environments have been established 
+as the most used interface between the actual application and the RL algorithm.
+
+**Deepbots is a framework which follows the gym interface logic and bridges the 
+gap between the gym environment and the simulator to enable you to easily 
+create custom RL environments in Webots.**
 
 ## Installation
 
@@ -28,7 +31,7 @@ logic in order to be used by Webots applications.
    refer to
    [Using Python](https://cyberbotics.com/doc/guide/using-python#introduction)
    to select the proper Python version for your system)
-3. Follow the [Using Python](https://cyberbotics.com/doc/guide/using-python)
+3. Refer to the [Using Python](https://cyberbotics.com/doc/guide/using-python)
    guide provided by Webots
 4. Webots provides a basic code editor, but if you want to use
    [PyCharm](https://www.jetbrains.com/pycharm/) as your IDE refer to
@@ -38,8 +41,10 @@ logic in order to be used by Webots applications.
 You will probably also need a backend library to implement the neural networks,
 such as [PyTorch](https://pytorch.org/) or
 [TensorFlow](https://www.tensorflow.org/). Deepbots interfaces with RL agents
-using the OpenAI gym logic, so it can work with any backend library you choose
-to implement the agent with and any agent that already works with gym.
+using the gym logic, so it can work with any backend library you choose
+to implement the agent with and any agent that already works with gym, such
+as [stable-baselines3](https://github.com/DLR-RM/stable-baselines3) 
+implementations.
 
 ### Install deepbots
 
@@ -48,6 +53,9 @@ Deepbots can be installed through the package installer
 
 `pip install deepbots`
 
+If you encounter [this](https://github.com/aidudezzz/deepbots/issues/143) 
+issue please use `pip install setuptools==65.5.0` before installing deepbots.
+
 ## Official resources
 
 - On
@@ -56,6 +64,8 @@ Deepbots can be installed through the package installer
 - On [the deepworlds repository](https://github.com/aidudezzz/deepworlds) you
   can find examples of deepbots being used. <br>Feel free to contribute your
   own!
+- On [the deepbots documentation site](https://deepbots.readthedocs.io/)
+  you can find the framework's documentation
 
 ## Citation
 
@@ -79,154 +89,6 @@ https://link.springer.com/chapter/10.1007/978-3-030-49186-4_6
 
 ```
 
-## How it works
-
-First of all let's set up a simple glossary:
-
-- `World`: Webots uses a tree structure to represent the different entities in
-  the scene. The World is the root entity which contains all the
-  entities/nodes. For example, the world contains the Supervisor and Robot
-  entities as well as other objects which might be included in the scene.
-
-- `Supervisor`: The Supervisor is an entity which has access to all other
-  entities of the world, while having no physical presence in it. For example,
-  the Supervisor knows the exact position of all the entities of the world and
-  can manipulate them. Additionally, the Supervisor has the Supervisor
-  Controller as one of its child nodes.
-
-- `Supervisor Controller`: The Supervisor Controller is a python script which
-  is responsible for the Supervisor. For example, in the Supervisor Controller
-  script the distance between two entities in the world can be calculated.
-
-- `Robot`: The Robot is an entity that represents a robot in the world. It
-  might have sensors and other active components, like motors, etc. as child
-  entities. Also, one of its children is the Robot Controller. For example,
-  [epuck](https://cyberbotics.com/doc/guide/epuck) and
-  [TIAGo](https://cyberbotics.com/doc/guide/tiago-iron) are robots.
-
-- `Robot Controller`: The Robot Controller is a python script which is
-  responsible for the Robot's movement and sensors. With the Robot Controller
-  it is possible to observe the world and act accordingly.
-- `Environment`: The Environment is the interface as described by the OpenAI
-  gym. The Environment interface has the following methods:
-
-  - `get_observations()`: Return the observations of the robot. For example,
-    metrics from sensors, a camera image etc.
-
-  - step(action): Each timestep, the agent chooses an action, and the
-    environment returns the observation, the reward and the state of the
-    problem (done or not).
-
-  - `get_reward(action)`: The reward the agent receives as a result of their
-    action.
-  - `is_done()`: Whether it’s time to reset the environment. Most (but not all)
-    tasks are divided up into well-defined episodes, and done being True
-    indicates the episode has terminated. For example, if a robot has the task
-    to reach a goal, then the done condition might happen when the robot
-    "touches" the goal.
-  - `reset()`: Used to reset the world to the initial state.
-
-In order to set up a task in Deepbots it is necessary to understand the
-intention of the OpenAI gym environment. According to the OpenAI gym
-documentation, the framework follows the classic “agent-environment loop”.
-"Each timestep, the agent chooses an `action`, and the environment returns an
-`observation` and a `reward`. The process gets started by calling `reset()`,
-which returns an initial `observation`."
-
-<p align="center">
-    <img src="https://raw.githubusercontent.com/aidudezzz/deepbots/dev/doc/img/agent_env_loop.svg">
-</p>
-
-Deepbots follows this exact agent-environment loop with the only difference
-being that the agent, which is responsible to choose an action, runs on the
-Supervisor and the observations are acquired by the robot. The goal of the
-deepbots framework is to hide this communication from the user, especially from
-those who are familiar with the OpenAI gym environment. More specifically,
-`SupervisorEnv` is the interface which is used by the Reinforcement Learning
-algorithms and follows the OpenAI Gym environment logic. The Deepbots framework
-provides different levels of abstraction according to the user's needs.
-Moreover, a goal of the framework is to provide different wrappers for a wide
-range of robots.
-
-Deepbots also provides a default implementation of the `reset()` method,
-leveraging Webots' built-in simulation reset functions, removing the need for
-the user to implement reset procedures for simpler use-cases. It is always
-possible to override this method and implement any custom reset procedure, as
-needed.
-
-#### Emitter - receiver scheme
-
-Currently, the communication between the `Supervisor` and the `Robot` is
-achieved via an `emitter` and a `receiver`. Separating the `Supervisor` from
-the `Robot`, deepbots can fit a variety of use-cases, e.g. multiple `Robots`
-collecting experience and a `Supervisor` controlling them with a single agent.
-The way Webots implements `emitter`/`receiver` communication requires messages
-to be packed and unpacked, which introduces an overhead that becomes
-prohibiting in use-cases where the observations are high-dimensional or long,
-such as camera images. Deepbots provides another partially abstract class that
-combines the `Supervisor` and the `Robot` into one controller and circumvents
-that issue, while being less flexible, which is discussed
-[later](#combined-robot-supervisor-scheme).
-
-<p align="center">
-    <img src="https://raw.githubusercontent.com/aidudezzz/deepbots/dev/doc/img/deepbots_overview.png">
-</p>
-
-On one hand, the `emitter` is an entity which is provided by Webots, that
-broadcasts messages to the world. On the other hand, the `receiver` is an
-entity that is used to receive messages from the `World`. Consequently, the
-agent-environment loop is transformed accordingly. Firstly, the `Robot` uses
-its sensors to retrieve the observation from the `World` and in turn uses the
-`emitter` component to broadcast this observation. Secondly, the `Supervisor`
-receives the observation via the `receiver` component and in turn, the agent
-uses it to choose an action. It should be noted that the observation the agent
-uses might be extended from the `Supervisor`. For example, a model might use
-LiDAR sensors installed on the `Robot`, but also the Euclidean distance between
-the `Robot` and an object. As it is expected, the `Robot` does not know the
-Euclidean distance, only the `Supervisor` can calculate it, because it has
-access to all entities in the `World`.
-
-You can follow the 
-[emitter-receiver scheme tutorial](https://github.com/aidudezzz/deepbots-tutorials/blob/master/emitterReceiverSchemeTutorial/README.md)
-to get started and work your way up from there.
-
-<p align="center">
-    <img src="https://raw.githubusercontent.com/aidudezzz/deepbots/dev/doc/img/workflow_diagram.png">
-</p>
-
-#### Combined Robot-Supervisor scheme
-
-As mentioned earlier, in use-cases where the observation transmitted between
-the `Robot` and the `Supervisor` is high-dimensional or long, e.g. high
-resolution images taken from a camera, a significant overhead is introduced.
-This is circumvented by inheriting and implementing the partially abstract
-`RobotSupervisor` that combines the `Robot controller` and the
-`Supervisor Controller` into one, forgoing all `emitter`/`receiver`
-communication. This new controller runs on the `Robot`, but requires
-`Supervisor` privileges and is limited to one `Robot`, one `Supervisor`.
-
-You can follow the 
-[robot-supervisor scheme tutorial](https://github.com/aidudezzz/deepbots-tutorials/tree/master/robotSupervisorSchemeTutorial)
-to get started and work your way up from there. We recommended this
-tutorial to get started with deepbots.
-
-### Abstraction Levels
-
-The deepbots framework has been created mostly for educational purposes. The
-aim of the framework is to enable people to use Reinforcement Learning in
-Webots. More specifically, we can consider deepbots as a wrapper of Webots
-exposing an OpenAI gym style interface. For this reason there are multiple
-levels of abstraction. For example, a user can choose if they want to use CSV
-`emitter`/`receiver` or if they want to make an implementation from scratch. In
-the top level of the abstraction hierarchy is the `SupervisorEnv` which is the
-OpenAI gym interface. Below that level there are partially implemented classes
-with common functionality. These implementations aim to hide the communication
-between the `Supervisor` and the `Robot`, as described in the two different
-schemes ealier. Similarly, in the `emitter`/`receiver` scheme the `Robot` also
-has different abstraction levels. According to their needs, users can choose
-either to process the messages received from the `Supervisor` themselves or use
-the existing implementations.
-
 ### Acknowledgments
 
 This project has received funding from the European Union's Horizon 2020

deepbots/supervisor/controllers/deepbots_supervisor_env.py

@@ -50,7 +50,7 @@ def reset(self):
         Default, problem-agnostic, implementation of reset method,
         using Webots-provided methods.
 
-        *Note that this works properly only with Webots versions >R2020b
+        \*Note that this works properly only with Webots versions >R2020b
         and must be overridden with a custom reset method when using
         earlier versions. It is backwards compatible due to the fact
         that the new reset method gets overridden by whatever the user

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/_static/css/baselines_theme.css

@@ -0,0 +1,63 @@
+/* Taken from https://github.com/DLR-RM/stable-baselines3/blob/master/docs/_static/css/baselines_theme.css */
+
+/* Main colors  adapted from pytorch doc */
+:root{
+  --main-bg-color: #343A40;
+  --link-color: #FD7E14;
+}
+
+/* Header fonts y */
+h1, h2, .rst-content .toctree-wrapper p.caption, h3, h4, h5, h6, legend, p.caption {
+    font-family: "Lato","proxima-nova","Helvetica Neue",Arial,sans-serif;
+}
+
+
+/* Docs background */
+.wy-side-nav-search{
+  background-color: var(--main-bg-color);
+}
+
+/* Mobile version */
+.wy-nav-top{
+  background-color: var(--main-bg-color);
+}
+
+/* Change link colors (except for the menu) */
+a {
+    color: var(--link-color);
+}
+
+a:hover {
+    color: #4F778F;
+}
+
+.wy-menu a {
+    color: #b3b3b3;
+}
+
+.wy-menu a:hover {
+    color: #b3b3b3;
+}
+
+a.icon.icon-home {
+    color: #b3b3b3;
+}
+
+.version{
+    color: var(--link-color) !important;
+}
+
+
+/* Make code blocks have a background */
+.codeblock,pre.literal-block,.rst-content .literal-block,.rst-content pre.literal-block,div[class^='highlight'] {
+        background: #f8f8f8;;
+}
+
+/* Change style of types in the docstrings .rst-content .field-list */
+.field-list .xref.py.docutils, .field-list code.docutils, .field-list .docutils.literal.notranslate
+{
+  border: None;
+  padding-left: 0;
+  padding-right: 0;
+  color: #404040;
+}