- Task Description
- Feature Vector Format
- Action Label Response and Feedback
- Performance
- Novelty Indicator
- Novelty Characterization
- Sample (Mock) Novelty
- Revealed Novelty
- Frequently Asked Questions
The novelty level-0 Smart Environment task is for the agent to recognize the current activity of the inhabitant in the environment. The agent receives timestamped sensor data and responds with one of 11 possible activities: wash dishes, relax, personal hygiene, bed toilet transition, cook, sleep, take medicine, leave home, work, enter home, eat. The agent also receives feedback about their current performance, which is the percentage of correct classifications since the beginning of the episode. The inhabitant's behavior varies from day to day. An episode is one day's worth of data. Sensor vectors are generated when at least one sensor changes value, not at a fixed frequency. Below is a sample floorplan of one of the smart environments.
See the smartenv.json file for a precise specification of the domain, including ranges on sensor values.
The feature vector provides a value for each sensor and is sent in JSON format. There are four types of sensors: motion, motion area, light switch, and door. Each sensor is described by its ID and value. There may be more than one of each sensor type.
For example,
{
"time_stamp": 1597953858.2,
"motion_sensors": [
{ "id": "M001", "value": 0.0 },
{ "id": "M002", "value": 1.0 }
],
"motion_area_sensors": [
{ "id": "MA007", "value": 1.0 }
],
"door_sensors": [
{ "id": "D011", "value": 0.0 }
],
"light_switch_sensors": [
{ "id": "L021", "value": 1.0 },
{ "id": "L022", "value": 0.0 }
]
}
The first call will have an additional feature vector:
- Hints given pointers to the novelty used.
Full hint example:
"hint": {
"level": 1,
"entity": "sensor",
"attribute": "quantity",
"change": "decrease"
}
No hint example:
"hint": {
"level": None,
"entity": None,
"attribute": None,
"change": None
}
After each sensor vector, the agent responds with one of 11 possible activity classifications: wash_dishes, relax, personal_hygiene, bed_toilet_transition, cook, sleep, take_medicine, leave_home, work, enter_home, eat.
The agent is then provided with the correct activiy classification according to a budget. E.g., if the budget is 50%, then the correct classificaton is provided half the time. The budget is set within the novelty generator and made known to the TA2 team.
The current performance of the agent on the current episode is provided as feedback after each agent response and is sent in JSON format. For SmartEnv, performance is defined as classification accuracy so far, i.e. the number of correct predictions divided by the total number of predictions. The performance at the end of the episode is recorded as the performance for that entire episode.
After each sensor fecture vector, the novelty generator sends a novelty indicator, which indicates if the current episode is novel "true", not novel "false" (i.e., novelty level 0), or unknown "null". The novelty indicator will be the same for every turn during an episode.
At the end of each episode (train and test), the agent provides a novelty characterization, which includes: a list of novelty levels [0, 8] and their probabilities [0.0, 1.0], an entity characterization string, an attribute characterization string, and directional change string.
TA2s must report at least the novelty levels and their probabilities. Missing levels will be presumed to be 0.0.
It should look like:
"novelty_characterization": {
"level": [
{"level_number": int, "probability": float},
...,
{"level_number": int, "probability": float}
],
"entity": str,
"attribute": str,
"change": str
}
Available options for entity, attribute, and change:
- Entity: inhabitant, sensor, floorplan, room, timestamp, hour, day, month, year (+ Activity) (+Sensor)
- Activity: wash_dishes, relax, personal_hygiene, bed_toilet_transition, cook, sleep, take_medicine, leave_home, work, enter_home, eat
- Sensor: motion, area, door, light
- Attribute: quantity, variance, correlation, frequency
- Change: increase, decrease, increasing, decreasing, away <entity>, toward <entity>, shift
The SmartEnv novelty generator includes sample Phase 2 novelties for levels 1-5, also called Mock novelties. These are described below.
- Level 1: Some sensors turned off.
- Level 2: Inhabitant performs exact same behavior each day.
- Level 3: Activity not seen in pre-novelty, but seen in post-novelty.
- Level 4: Two sensor values always give same reading.
- Level 5: Second inhabitant simulated by overlaying copy of existing inhabitant.
- Level 6: Sensor values provided at random times.
- Level 7: Inhabitant spends more time outside of home.
- Level 8: Multiple days of data compressed into one.
Table of how hints are returned using hint level:
| HintLevel | NoveltyLevel | Entity | Attribute | Change |
|---|---|---|---|---|
| -1 | None | None | None | None |
| 0 | Given | None | None | None |
| 1 | Given | Given | None | None |
| 2 | Given | Given | Given | None |
| 3 | Given | Given | Given | Given |
Table of hints for mock novelties:
| Level | Entity | Attribute | Change |
|---|---|---|---|
| 1 | sensor | quantity | decrease |
| 2 | sensor | variance | decrease |
| 3 | activity | distribution | shift |
| 4 | sensor | correlation | increase |
| 5 | inhabitant | quantity | increase |
| 6 | sensor | frequency | shift |
| 7 | leave_home | quantity | increase |
| 7 | enter_home | quantity | increase |
| 8 | sensor | frequency | decrease |
-
Level 1 (Class): Time shifts to some time in the future.
- No Novelty: Episodes are contiguous days
- Easy: Novel episodes are ? days into the future
- Medium: Novel episodes are ? days into the future
- Hard: Novel episodes are ? days into the future
-
Level 2 (Attribute): Sensors inactive in pre-novelty become active post-novelty
- No Novelty: 0 inactive sensors become active
- Easy: 5 inactive sensors become active
- Medium: 10 inactive sensors become active
- Hard: All inactive sensors become active (15-20 depending on floorplan)
-
Level 3 (Representation): Partition floorplan into areas; all sensors in area have same value
- No novelty: # areas = # sensors
- Easy: Floorplan partitioned into 10 areas
- Medium: Floorplan partitioned into 8 areas
- Hard: Floorplan partitioned into 5 areas
-
Level 1 (Objects): Increase in available sensors
- No novelty: No additional sensors
- Easy: 5 additional sensors
- Medium: 10 additional sensors
- Hard: Rest 30 additional sensors
-
Level 2 (Agents): Time shifts to some time in the future
- No Novelty: Episodes are contiguous days
- Easy: Novel episodes are 30-60 days into the future
- Medium: Novel episodes are 120-150 days into the future
- Hard: Novel episodes are 260-300 days into the future
-
Level 3 (Actions): Different inhabitant in same floorplan
- No novelty: No change in inhabitant
- Easy: Small change in inhabitant behavior
- Medium: Medium change in inhabitant behavior
- Hard: Large change in inhabitant behavior
-
Level 4 (Relations): Sensors in one room turned off
- No novelty: No sensors turned off
- Easy: Study (5) sensors turned off in a room
- Medium: Living room (6) sensors turned off in a room
- Hard: Bedroom (7) sensors turned off in a room
-
Level 5 (Interactions): Second inhabitant visits the environment
- No novelty: No additional inhabitants
- Easy: ? additional inhabitants visit for duration ?
- Medium: ? additional inhabitants visit for duration ?
- Hard: ? additional inhabitants visit for duration ?
-
Level 6 (Rules): Temperature decrease causes motion sensors to trigger more often.
- No novelty: Normal motion sensor behavior.
- Easy: Small increase in motion sensor triggers.
- Medium: Medium increase in motion sensor triggers.
- Hard: Large increase in motion sensor triggers.
-
Level 7 (Goals): Inhabitant attempts to fool caregiver into thinking they went outside, but in fact they did not. This is based on fake leave_home and enter_home events inserted in the sensor stream, but activity label is prior activity.
- No novelty: Normal frequency of leave_home and enter_home events.
- Easy: Small increase in number of leave_home and enter_home events.
- Medium: Medium increase in number of leave_home and enter_home events.
- Hard: Large increase in number of leave_home and enter_home events.
-
Level 8 (Events): Multiple days worth of data combined into one (fast forward).
- No novelty: Episode data represents a normal 24-hour day.
- Easy: Novel episodes contain 1.5 days worth of data.
- Medium: Novel episodes contain 2 days worth of data.
- Hard: Novel episodes contain 3 days worth of data.
Coming soon...