About me

I'm Feng, a Software Engineer at AI Rudder, a startup in San Mateo, California, where we're dedicated to empowering LLMs (Large Language Models). At AI Rudder, We integrate with Text-to-Speech (TTS) and Automatic Speech Recognition (ASR) technologies to develop innovative voice bots for phone calls.

I received my master's degree in Computer Science from University of Illinois at Urbana-Champaign in Winter 2021. Before pursuing my master's degree, I served as a Research Intern at Westlake University, advised by Prof. Lanzhenzhong in the Deep Learning Lab. Prior to that, I received my bachelor's degree in Computer Science from Ohio State University in 2020.

My research interests are deeply rooted in unraveling the fundamental principles behind human language and communication. I am driven by the challenge of translating these insights into practical algorithms that address real-world challenges. This drive fuels my enthusiasm for life.

Publication

  • Toward the Limitation of Code-Switching in Cross-Lingual Transfer

    Y Feng, F Li, P Koehn EMNLP

    Multilingual pretrained models have shown strong cross-lingual transfer ability. Some works used code-switching sentences, which consist of tokens from multiple languages, to enhance the cross-lingual representation further, and have shown success in many zero-shot cross-lingual tasks. However, code-switched tokens are likely to cause grammatical incoherence in newly substituted sentences, and negatively affect the performance on token-sensitive tasks, such as Part-of-Speech (POS) tagging and Named-Entity-Recognition (NER). This paper mitigates the limitation of the code-switching method by not only making the token replacement but considering the similarity between the context and the switched tokens so that the newly substituted sentences are grammatically consistent during both training and inference. We conduct experiments on cross-lingual POS and NER over 30+ languages, and demonstrate the effectiveness of our method by outperforming the mBERT by 0.95 and original code-switching method by 1.67 on F1 scores.

    Paper Code

    December 2022

  • Learn To Remember: Transformer With Recurrent Memory For Document-Level Machine Translation

    Y Feng, F Li, Z Song, B Zheng, P Koehn NAACL

    The Transformer architecture has led to significant gains in machine translation. However, most studies focus on only sentence-level translation without considering the context dependency within documents, leading to the inadequacy of document-level coherence. Some recent research tried to mitigate this issue by introducing an additional context encoder or translating with multiple sentences or even the entire document. Such methods may lose the information on the target side or have an increasing computational complexity as documents get longer. To address such problems, we introduce a recurrent memory unit to the vanilla Transformer, which supports the information exchange between the sentence and previous context. The memory unit is recurrently updated by acquiring information from sentences, and passing the aggregated knowledge back to subsequent sentence states. We follow a two-stage training strategy, in which the model is first trained at the sentence level and then finetuned for document-level translation. We conduct experiments on three popular datasets for document-level machine translation and our model has an average improvement of 0.91 s-BLEU over the sentence-level baseline. We also achieve state-of-the-art results on TED and News, outperforming the previous work by 0.36 s-BLEU and 1.49 d-BLEU on average.

    Paper Code

    July 2022

What's New

Learn To Remember: Transformer With Recurrent Memory For Document-Level Machine Translation

The Transformer architecture has led to significant gains in machine translation. However, most studies focus on only sentence-level translation without considering the context dependency within documents, leading to the inadequacy of document-level coherence. Some recent research tried to mitigate this issue by introducing an additional context encoder or translating with multiple sentences or even the entire document. Such methods may lose the information on the target side or have an increasing computational complexity as documents get longer. To address such problems, we introduce a recurrent memory unit to the vanilla Transformer, which supports the information exchange between the sentence and previous context. The memory unit is recurrently updated by acquiring information from sentences, and passing the aggregated knowledge back to subsequent sentence states. We follow a two-stage training strategy, in which the model is first trained at the sentence level and then finetuned for document-level translation. We conduct experiments on three popular datasets for document-level machine translation and our model has an average improvement of 0.91 s-BLEU over the sentence-level baseline. We also achieve state-of-the-art results on TED and News, outperforming the previous work by 0.36 s-BLEU and 1.49 d-BLEU on average.

Paper Code