diff --git a/content/chapters/02_dl_basics/02_01_optimization.md b/content/chapters/02_dl_basics/02_01_optimization.md deleted file mode 100644 index db2760a..0000000 --- a/content/chapters/02_dl_basics/02_01_optimization.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -title: "Chapter 02.01: Optimization" -weight: 2001 - ---- -This chapter discusses optimization and the optimization algorithm gradient descent. The variants of stochastic gradient descent are commonly used for deep learning. - - - -### Lecture slides - -{{< pdfjs file="https://github.com/slds-lmu/lecture_dl4nlp/blob/main/slides/chapter2-deeplearningbasics/slides-21-optimization.pdf" >}} - diff --git a/content/chapters/02_dl_basics/02_01_rnn.md b/content/chapters/02_dl_basics/02_01_rnn.md new file mode 100644 index 0000000..76e9292 --- /dev/null +++ b/content/chapters/02_dl_basics/02_01_rnn.md @@ -0,0 +1,13 @@ +--- +title: "Chapter 02.01: Recurrent Neural Networks" +weight: 2001 + +--- +This chapter introduces Recurrent Neural Networks in the context of Language Modelling and discusses different types of RNNs, such as LSTMs and Bidirectional RNNs. + + + +### Lecture slides + +{{< pdfjs file="https://github.com/slds-lmu/lecture_dl4nlp/blob/main/slides/chapter02-deeplearningbasics/slides-21-rnn.pdf" >}} + diff --git a/content/chapters/02_dl_basics/02_02_attention.md b/content/chapters/02_dl_basics/02_02_attention.md new file mode 100644 index 0000000..f9de821 --- /dev/null +++ b/content/chapters/02_dl_basics/02_02_attention.md @@ -0,0 +1,12 @@ +--- +title: "Chapter 02.02 Attention" +weight: 2002 +--- + +This chapter provides a first introduction to the Attention mechanism as a way to model long range dependencies. + + + +### Lecture slides + +{{< pdfjs file="https://github.com/slds-lmu/lecture_dl4nlp/blob/main/slides/chapter02-deeplearningbasics/slides-22-attention.pdf" >}} \ No newline at end of file diff --git a/content/chapters/02_dl_basics/02_02_dnns_activations.md b/content/chapters/02_dl_basics/02_02_dnns_activations.md deleted file mode 100644 index 19d9761..0000000 --- a/content/chapters/02_dl_basics/02_02_dnns_activations.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -title: "Chapter 02.02 Deep Feedforward Networks" -weight: 2002 ---- - -This chapter discusses Deep Forward Networks, the simplest type of neural networks and some common activation functions, also called non-linearities. - - - -### Lecture slides - -{{< pdfjs file="https://github.com/slds-lmu/lecture_dl4nlp/blob/main/slides/chapter2-deeplearningbasics/slides-22-dnns_activations.pdf" >}} \ No newline at end of file diff --git a/content/chapters/02_dl_basics/02_03_backpropagation.md b/content/chapters/02_dl_basics/02_03_backpropagation.md deleted file mode 100644 index a7aeacd..0000000 --- a/content/chapters/02_dl_basics/02_03_backpropagation.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -title: "Chapter 02.03: Backpropagation" -weight: 2003 ---- -The ability to scale neural networks is a key criterion for the success of modern neural networks. The backpropagation algorithm powers the efficient computation of the gradients. - - - -### Lecture slides - -{{< pdfjs file="https://github.com/slds-lmu/lecture_dl4nlp/blob/main/slides/chapter2-deeplearningbasics/slides-23-backpropagation.pdf" >}} - diff --git a/content/chapters/02_dl_basics/02_03_elmo.md b/content/chapters/02_dl_basics/02_03_elmo.md new file mode 100644 index 0000000..26797fb --- /dev/null +++ b/content/chapters/02_dl_basics/02_03_elmo.md @@ -0,0 +1,12 @@ +--- +title: "Chapter 02.03: ELMo" +weight: 2003 +--- +In this chapter we introduce ELMo, a modelling approach, that enables us to contextualize word embeddings. + + + +### Lecture slides + +{{< pdfjs file="https://github.com/slds-lmu/lecture_dl4nlp/blob/main/slides/chapter02-deeplearningbasics/slides-23-elmo.pdf" >}} + diff --git a/content/chapters/02_dl_basics/02_04_tokenization.md b/content/chapters/02_dl_basics/02_04_tokenization.md new file mode 100644 index 0000000..69afea9 --- /dev/null +++ b/content/chapters/02_dl_basics/02_04_tokenization.md @@ -0,0 +1,11 @@ +--- +title: "Chapter 02.04 Revisiting words: Tokenization" +weight: 2004 +--- +In order to feed text data into a model we have to tokenize it first. This chapter discusses various types of text tokenization. + + + +### Lecture slides + +{{< pdfjs file="https://github.com/slds-lmu/lecture_dl4nlp/blob/main/slides/chapter02-deeplearningbasics/slides-24-tokenization.pdf" >}} diff --git a/content/chapters/02_dl_basics/02_regularization.md b/content/chapters/02_dl_basics/02_regularization.md deleted file mode 100644 index bd13827..0000000 --- a/content/chapters/02_dl_basics/02_regularization.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -title: "Chapter 02.04 Regularization" -weight: 2004 ---- -Regularization provides the possibility to guide the learning process. - - - -### Lecture slides - -{{< pdfjs file="https://github.com/slds-lmu/lecture_dl4nlp/blob/main/slides/chapter2-deeplearningbasics/slides-24-regularization.pdf" >}} diff --git a/content/chapters/02_dl_basics/_index.md b/content/chapters/02_dl_basics/_index.md index a728fb7..b1e2479 100644 --- a/content/chapters/02_dl_basics/_index.md +++ b/content/chapters/02_dl_basics/_index.md @@ -1,10 +1,6 @@ --- title: "Chapter 2: Deep Learning Basics" --- -This chapter gives a quick introduction to the basic concepts of deep learning, such as optimization, simple Feedforward networks, backpropagation and regularization. A comprehensive introduction is given in [1]. +This chapter gives a quick introduction to the basic concepts of deep learning in the context of NLP, such as RNN, attention, ELMo and tokenization. - - -### References - -- [1] [Goodfellow et al., 2016](https://www.deeplearningbook.org/) + \ No newline at end of file