First Pass of MAIRL

[lalitlal]

Major In progress

1. Create Reward Fn and Value Fn in Discriminator [DONE]

• Inside init() or forward() -> added inside forward()

2. [discriminator_irl.py] - modified discriminator.py [DONE]

• Calculate log_p_tau, log_q_tau, log_pq for external losses

3. [mgail.py] - modify how our discriminator is changed [DONE]

• See Fixed std for policy or learned std? #2. Discriminator section

4. [mgail.py] - modify al_loss to take into account new [DONE] discriminator output

5. Modify expert data to include expert action probs [TODO]

• Need this for lprobs

6. Modify ER to account for new field 'action_probs' [TODO]

TO DO

• Testing that we didn't break existing code
• Testing the MAIRL actually compiles
• Testing that MAIRL actually works as expected