The Forward Forward algorithm
A potential replacement / augment to the gradient based backpropagation algorithm used to train neural networks. Building test benches to test the algorithm on various datasets and comparing it’s performance to that of backpropagation is in progress, but while a CNN was easy pickings, building an RNN is turning outto be a bit of a challenge.
Yeah, just understanding the algorithm’s math is a challenge in itself. I’m not even going to try and explain it here. I’ll just link the paper and let you figure it out for yourself. You can find the original paper on ArXiv here or just download it by clicking here
In essence, instead of using the gradient all the way upto the first layer to update the weights, we basically train the network twice in one iteration - once to train it to positively identify the correct class and once to train it to negatively identify the incorrect class. The weights are then updated using the difference between the two. This is a very crude explanation and I’m not doing the algorithm justice, but I’m still trying to wrap my head around it myself so I’ll leave it at that.
In a much more rougher sense, train it using a positive goodness and negative goodness function and then update the weights using the difference between the two. Minimise the difference between the two goodness functions and you have a trained network.
I’ll be writing a detailed post on this soon, so stay tuned for that.
HyperLogLog
A probabilistic cardinality estimator. I went on a Wikipedia Rabbit hole and ended up reading about this and other probabilistic data structures. Figured out the math to a degree and now I’m just fascinated by the intellect of the people who came up with this.
I started exploring this when I was trying to figure out how Google Analytics works. I was trying to figure out how they estimate the number of unique visitors to a website ,but was stumped. But that’s when I was giving a friend of mine a quick rundown on BigData for an interview and the Flajolet Martin algorithm came back to me. I had read about it a while back and had forgotten about it. Another internet rabbit hole later is when I came across HyperLogLog.
Meta actually implements HyperLogLog in ‘Presto’, their distributed query engine. Would be cool to work on something like that someday.