100 Days of Machine Learning Code Day 14

Day 14: MNIST dataset for handwriting classification  using Tensorflow 

Q. Whats MNIST Dataset  ?

A. The MNIST dataset is an image dataset of handwritten digits made available by Yann LeCun et. al. here. It has has 60,000 training images and 10,000 test images, each of which are grayscale 28 x 28 sized images. It is a good beginner’s dataset to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on pre-processing and formatting

Key Takeaways  :  

• Sequential: That defines a SEQUENCE of layers in the neural network
• Flatten: Remember earlier where our images were a square, when you printed them out? Flatten just takes that square and turns it into a 1 dimensional set.
• Dense: Adds a layer of neurons
• Relu effectively means "If X>0 return X, else return 0" -- so what it does it it only passes values 0 or greater to the next layer in the network.
• Softmax takes a set of values, and effectively picks the biggest one, so, for example, if the output of the last layer looks like [0.1, 0.1, 0.05, 0.1, 9.5, 0.1, 0.05, 0.05, 0.05], it saves you from fishing through it looking for the biggest value, and turns it into [0,0,0,0,1,0,0,0,0] -- The goal is to save a lot of coding!

Revision from Day 5

Q .What will happen if we change the Number Of Dense Neurons (512) in our Neural Network  ?

Ans . by adding more Neurons we have to do more calculations, slowing down the process, but in this case they have a good impact -- we do get more accurate. That doesn't mean it's always a case of 'more is better', you can hit the law of diminishing returns very quickly

Q. What would happen if you remove the Flatten() layer. Why do you think that's the case?

Ans. You get an error about the shape of the data. It may seem vague right now, but it reinforces the rule of thumb that the first layer in your network should be the same shape as your data. Right now our data is 28x28 images, and 28 layers of 28 neurons would be infeasible, so it makes more sense to 'flatten' that 28,28 into a 784x1. Instead of wriitng all the code to handle that ourselves, we add the Flatten() layer at the begining, and when the arrays are loaded into the model later, they'll automatically be flattened for us.

Q .Consider the final (output) layers. Why are there 10 of them? What would happen if you had a different amount than 10?

Ans. You get an error as soon as it finds an unexpected value. Another rule of thumb -- the number of neurons in the last layer should match the number of classes you are classifying for. In this case it's the digits 0-9, so there are 10 of them, hence you should have 10 neurons in your final layer.

Q. Consider the effects of additional layers in the network. What will happen if you add another layer between the one with 512 and the final layer with 10.

Ans .There isn't a significant impact -- because this is relatively simple data. For far more complex data including color images to be classified , extra layers are often necessary.

Q. Consider the impact of training for more or less epochs. Why do you think that would be the case?

Ans .Try 15 epochs -- you'll probably get a model with a much better loss than the one with 5 Try 30 epochs -- you might see the loss value stops decreasing, and sometimes increases. This is a side effect of something called 'overfitting' which you can learn about [somewhere] and it's something you need to keep an eye out for when training neural networks. There's no point in wasting your time training if you aren't improving your loss, right! :)

Source :  Notebook 

My Solution with Callback  : In Coursera Jupyter Notebook 

My Solution without callback: In Google Colab Notebook

100 Days of Machine Learning Code Day 13

n-gram

Day 13 : Making my own  string processing code to generate the N-gram probability table for the given sequence of strings , But why N-gram all of a sudden in a ML learning quest  ?  

Q. Why N-gram though?

ans . Well, in Natural Language Processing, or NLP for short, n-grams are used for a variety of things. Some examples include auto completion of sentences (such as the one we see in Gmail these days), auto spell check (yes, we can do that as well), and to a certain extent, we can check for grammar in a given sentence.

Q .N-gram Probabilities ?

ans .Let’s take the example of a sentence completion system. This system suggests words which could be used next in a given sentence. Suppose I give the system the sentence “Thank you so much for your” and expect the system to predict what the next word will be. Now you and me both know that the next word is “help” with a very high probability. But how will the system know that? 

 When we’re building an NLP model for predicting words in a sentence, the probability of the occurrence of a word in a sequence of words is what matters. And how do we measure that? Let’s say we’re working with a bigram model here, and we have the following sentences as the training corpus:

1. Thank you so much for your help.
2. I really appreciate your help.
3. Excuse me, do you know what time it is?
4. I’m really sorry for not inviting you.
5. I really like your watch.

Let’s suppose that after training our model with this data, I want to write the sentence “I really like your garden.” Now because this is a bigram model, the model will learn the occurrence of every two words, to determine the probability of a word occurring after a certain word. For example, from the 2nd, 4th, and the 5th sentence in the example above, we know that after the word “really” we can see either the word “appreciate”, “sorry”, or the word “like” occurs. So the model will calculate the probability of each of these sequences.

Suppose we’re calculating the probability of word “w1” occurring after the word “w2,” then the formula for this is as follows:

count(w2 w1) / count(w2)

count(really like) / count(really)
= 1 / 3
= 0.33

Source PDF : https://web.stanford.edu/~jurafsky/slp3/3.pdf

My Jupyter Noteboook Link  :  Notebook

100 Days of Machine Learning Code Day 12

Day 12 Exploring NLP , We all are familiar with Siri, Cortana , Google Asistant  , Alexa  but being a beginner scratching the surface and starting with NLP is way slower in academic curiculum with majority of part being understanding parts of speech  , speech tagging, finding out and correcting sentence structure with even find finding words with similar word ending (memorising past participle of words in masters seems way over the top) the algorithms on the other hand are intersting such has - "Levenshtein distance"  which is used in things like autocorrect and word prediction in smartphone keyboards  also learnt about how the POS tagging leads to intent classification  (intent classification is major step in making smart assistant skills in Alexa or Google Assistant ) so searched around on more on Implementation updating keyword dictionary on terms like BERT  , new Nvidia Jarvis for recent Nvidia GTC 2020 , from both programming standpoint and NLP ML learning , the minimum edit distance question does make it experience so made a python program in colab for calculating the Minimum edit distance 

Here is my Notebook for calculating minimum edit distance  

100 Days of Machine Learning Code Day 11

Day 11 : Finishing off with the  Week 1 of  "Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization" which included not 1 but 3 assignment notebooks on the topics of Initialization,Regularization  and Gradient checking

1. Topic  : Initalization , My notebook 

• Understand that different initialization methods and their impact on your model performance
• Implement zero initialization and and see it fails to "break symmetry",
• Recognize that random initialization "breaks symmetry" and yields more efficient models,
• use both random initialization and scaling to get even better training performance on model.

2.  Topic : Regularization , My notebook 

• different regularization methods that could help  model.
• Implement dropout and see it work on data.
• Recognize that a model without regularization gives  a better accuracy on the training set but nor necessarily on the test set.
• Understand that you could use both dropout and regularization on your model.

3. Topic : Gradient Checking , My Notebook

• Implement gradient checking from scratch.
• how to use the difference formula to check your backpropagation implementation.
• Recognizing that  backpropagation algorithm should give you similar results as the ones you got by computing the difference formula.
• how to identify which parameter's gradient was computed incorrectly.

100 Days of Machine Learning Code Day 10

Day 10  : Hello world in Pytorch , but before that difference we need to know PyTorch vs TensorFlow 

PyTorch	Tensorflow
Based on the Torch library	Based on Theano library
Distinguish Feature : Support for CUDA	Distinguish Feature : TensorBoard
Debugging :Dynamic computational process.	Debugging : Requires the TensorFlow debugger tool
PyTorch offers an advantage with its dynamic nature of creating the graphs. The graphs are built, interpreting the line of code corresponding to that particular aspect of the graph. However, in the case of TensorFlow, as the construction is static and the graph is required to go through compilation and then executed on execution engine.	Tensorflow works on a static graph concept that means the user first has to define the computation graph of the model and then run the ML model, whereas PyTorch believes in a dynamic graph that allows defining/manipulating the graph on the go.
PyTorch, on the other hand, has fewer features comparatively.	TensorFlow supports a higher level of functionality and gives a broad spectrum of options to work with by providing certain operations like: Flipping a tensor along with dimension Checking the Tensor for infinity and NaN Providing support for fast Fourier transforms Uses a package named contrib, for the creation of models.

TensorFlow Graph Explained : Video 

Here is my execution in Google colab   , After surfing around for Tensorflow vs Pytorch  here are my conclusions as beginner :

• Both are Widely used in Industry with majority of Google Projects in Tensorflow , and Tesla and Facebook other Deep Learning Implementation in pytorch
• Tensorflow is more easy to use and deploy in production and even mobile devices and app
• Static and Dynamic graph computation is a key differnce in basic working
• Both are equally powerful and with one having some advantage over the other  
• Tensorflow can utlize Google TPU in cloud where as Pytorch support CUDA core

One of the things that caught my attention is Specialised hardware for ML purpose like Intel Movidious , Google Coral  Chips and other specialised Hardware available in market , so will search more on that 

PS: University Decided to conduct the entire remaining semster Online so to make the best use of the daily routine will try to combine the theory lectures codes also in daily implementation , also checkout the videos on how Tesla uses Pytorch for Self Driving by Andrej Karpathy