Ep. 12 - Machine Learning and Speech Recognition with Kai-Fu Lee

Thank you, Tom. Great to be here.

Sure. So I got just excited about AI when I entered college in Columbia University in '79 and I was mentored by a number of professors. And I just thought AI was really the final step for humans to understand ourselves and how our brain works and whether we can build something half as smart as us. And that was going to be my life's dream. And with one of my advisers, John Kender, who was a CMU graduate taught at Columbia, he recommended me to CMU and I was so lucky I got in. And then CMU had this matching process where I got to hear every professor and I was really blown away by Raj Reddy who described his vision of the future and with his confidence and passion I thought this was it. I considered a number of areas in machine learning but speech recognition appealed to me the most because at the time compute was still very very slow and expensive and I didn't think something as large as general natural language understanding or computer vision for that matter was really possible for me to build something that would be an important step forward. Speech recognition frankly just felt like a more tractable problem because it's converting speech to text in at the time a limited domain. And then I chose Dr. Raj Reddy to be my professor and he taught me all about speech recognition, signal processing. And then at the time the speech group run by Raj was in the process of getting a large DARPA grant. It was to create a speaker independent continuous speech recognition system for a domain and that at the time was already an unsolved problem but it didn't seem so intractable so I entered into it. The primary machine learning method at the time was expert systems so that was the way the DARPA project overall was structured. It was built around another amazing computer scientist, Victor Zue, who could read the spectrogram. So he could look at the visual representation and read out what the person was saying. So the belief was that the spectrogram contained enough information if a human could read it then so can a machine. So that was kind of the overall direction of the DARPA project. A collaboration by a number of universities with CMU and MIT being the leading universities. And during my studies I ran into another amazing person, Dr. Peter Brown. He was I think two years ahead of me, a grad student, but he was basically working mostly at IBM under the supervision of Fred Jelinek and Bob Mercer and they were the ones who really were the first to use hidden Markov models. But IBM was not very open in describing all the ways in which they did things. And also I think people just felt that wasn't the best approach. So actually the mainstream approach was expert systems. The old approach was dynamic programming. That was the known proven approach invented by a Japanese scientist named Dr. Itakura. That was the known approach with some tiny commercial success. And then the DARPA approach was the next one is expert systems. But I got introduced luckily through Peter and with support from Fred and Bob that he would mentor me. I think what they wanted to see was just to be nice to a young guy and maybe when he graduates he'll join IBM. I think that's what they had in mind. Peter obviously didn't tell me any of what IBM was doing. But he taught me about hidden Markov models. Hidden Markov models are basically a form of neural networks but simplified for time series. So it's basically left to right. It has the ability to do very fast decoding using the Viterbi algorithm which is also a more generalized form of dynamic programming or dynamic time warping. So I felt that was the one I want to try. So I learned from Peter. I built some simple programs and then one day I just got enough courage to go to Raj Reddy to say that I love you as my adviser. I love speech recognition but can I try something different using hidden Markov models which is stochastic machine learning. But I would provide a different way to try the DARPA problem. It might or might not work but it would just be me. And later one or two other students joined me but at the time it was just me and Raj said something really amazing which was I don't agree with you but I will support you and I think that is a level of magnanimity, generosity and a way of management that I've learned a lot from. That in a scientific exploration you cannot just say this is the way. You have to let people pursue what they're good at and what their passion is. So he not only let me do the speech recognition my way but he asked me what resources I needed and I said well I need a lot of data because Fred Jelinek was famous for the quote that there's no data like more data. So I became a strong believer in that and that is still true today. So Raj went to DARPA and said well we need to get the National Institute of Standards to collect a bunch more data and let everybody use it. Expert system approach needs it too. And I was also very very lucky that with the DARPA grant the speech group at CMU was probably one of the most compute-rich speech groups in the world. A lot of some workstations were bought. They were SPARCstations really the most efficient for CPUs, not GPUs, but still the most efficient at the time. And the people building expert systems just used it to label spectrograms and then the expert system wasn't very compute intensive. So I got to use all the leftover cycles from the SPARCstations and there were more cycles left over than used. So I had at my disposal about I think about 20 SPARCstations which was phenomenal compute power. So with Raj's support I had Peter as my mentor, hidden Markov models, Raj as my sponsor with his wisdom and strategy and encouragement and equally importantly data and compute and that's how it began.

Sure. I think the state-of-the-art prior to my PhD work was IBM which built a so-called dictation system and which they also sold a little bit and also there was a company called Dragon also built by two CMU graduates, Jim and Janet Baker. And both systems pursued the dictation, their approach was the application and use case and people who would pay money are those who are unable to type. Namely people who were somehow disabled and then unable to type for whatever reason but clearly the technology could not handle the transcription like we are now getting used to. So they made two very very clever but very constraining requirements. One was each person has to train the speech recognizer. So when you bought the speech recognizer, you have to read a script. The system would say you know read the following sentence and you would read it and it would then take hours to train the hidden Markov model to recognize yours and only your speech and that was called speaker dependent speech recognition. But that wasn't good enough because continuous speech decoding was too large a problem even with Viterbi decoding. So they made another constraint which was you only could speak like this. Then what the system does is it takes the pause as a definitive breakpoint and recognize the next word so that your branching factor if your vocabulary was let's say 5,000 words the system only had to do okay I detected a pause the next word is going to be one of the next 5,000 given the previous word is recognized what might the probability be given both the acoustic evidence hidden Markov model recognition results but also language model which was what was the likelihood of the next word given the one or two previous words that were spoken and recognized. So that was one path. So it's called speaker dependent isolated dictation. That's one. The other that was also commercially sold was what was done by Bell Labs and because they were in the business of recognizing phone numbers. There was no hope of really recognizing anything complex because the telephone had a narrower bandwidth and also they could not do speaker dependent. If someone called up and the system said press one to ask about directions, press two to make a deposit, press three to be connected to operator. That got changed to press or say one to connect to sales, press or say two to get directions and so on. So they were able to add press or say. So they were able to make speaker independent but the system only did digits. So it had a vocabulary of 10. That was another way to constrain the problem. It could listen to anybody's speech. And the commercial systems generally also did isolated words even though there was beginnings of continuous digit recognition. So it was isolated digit recognition for speaker independent recognition and beginnings of speaker independent continuous digit recognition. That was what Bell Labs did and those were the state-of-the-art commercialized but with limited commercial benefit but nevertheless marginally useful to some people.

Yeah. So I also had to constrain the problem and the funding from DARPA came with a task. It was a resource allocation task. Basically something used in the government to ask about resources being used and applied and it basically had a context-free grammar. So it was continuous speech and also speaker independent that was the goal set out by DARPA with Raj's advice. So it was can expert systems deal with continuous speech speaker independent and large vocabulary. Large vocabulary being a thousand words, not quite dictation. And continuous speech being the way we speak, but only would work for things within a context-free grammar, which was a large branching factor, but not full dictation. And speaker independent was I think they had maybe around a hundred different speakers and it was important to prove speaker independent so that the speakers you test with must be different from the speakers you trained with. So those were the constraints set out and initially the accuracies were very low, 10 or 20% and then they got a little bit better but really never at a meaningful or useful level. I think they got maybe up to 50 or 60% word accuracy. So I decided I would basically take the same task they were doing because they had all the data and also they become a benchmark. So if I could beat that single-handedly that would be significant contribution and if I couldn't it would at least validate or invalidate an approach. And I came up with a number of ways that enhanced what I learned from IBM and Peter. At the time I don't think the IBM folks knew whether I would produce a great result or not because they were very much focused on speaker dependent. So it was meaningful to them to see if it would work but they didn't know I didn't know Raj didn't know but as I built a basic system it came out with about 70 or 80% accuracy. So that was already very very exciting. And on top of that I learned a lot from Victor Zue and his counterpart at CMU Ron Cole about the phonetic and acoustic aspects of speech. And I used some of their simple parts actually nothing complex. It was just that words are composed of phonemes which are sounds like b, a, t and these sounds these phonemes basically make up a word but some words might have multiple pronunciations like tomato tomato and furthermore phonemes don't exactly sound the same in different contexts so if I say bat and cat even though the a part sounds the same to our ears. But the B and C sound heavily affects the way a would be instantiated in the spectrogram. So that was one aspect of okay given that knowledge how can I build the hidden Markov model to deal with contextual influence across phonemes. That was one fundamental new idea that I had that other people did not have. And also I was lucky to have worked with a visiting researcher named Dr. Shikano from Japan and he was an expert in signal processing and he suggested Kai-fu we know that logarithmic representation of the spectrogram is much better for speech recognition so don't use the normal FFT spectrogram and he gave me his module which I plugged in which was called a mel-cepstral representation. So that also made a significant difference. And then lastly to build the language model I did not directly use the context-free grammar because that is very hard to, not hard but very time intensive to run with the Viterbi algorithm. So I thought why don't I just use bigrams and trigrams which is use the last two words to predict the next word and that was of course inherited from IBM. It's not a breakthrough for me but it was an important step. So and then there were a number of other small tweaks but it was a combination of Dr. Shikano's signal processing, my own ideas of how to improve the hidden Markov model structure to model the phonetics and acoustics, and using a statistical not a grammatical model to improve the speech recognition. So those were the things that I plugged in. And then I remember one Saturday I think I woke up and I saw that the system produced 96% accuracy and I was like blown away because before the best I could do was in the 80s and it turns out with these three main ideas plus a bunch of other tweaks I got up to 96%. And that was viewed as a big breakthrough at the time.