保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Data Science

"Chain-linking" NLP tasks With Wav2Vec2 & Transformers

Going straight from audio to a range of text-based NLP tasks such as translation, summarisation and sentiment analysis. Chua Chin Hon March 19, 20216 min read

Going straight from audio to a range of text-based NLP tasks such as translation, summarisation and sentiment analysis

Chart: Chua Chin Hon Chart: Chua Chin Hon The addition of the Wav2Vec2 model in Hugging Face's transformers library has been one of the more exciting developments in NLP in recent months. Until then, it wasn't easy to execute tasks like machine translation or sentiment analysis if you only had a long audio clip to work with. But now you can link up an interesting combination of NLP tasks in one go: transcribe the audio clip with Wav2Vec2, and then use a variety of transformer models to summarize or translate the transcript. The possible permutations and combinations of NLP tasks that you can link up is pretty mind boggling. Results could be patchy, of course. Some NLP tasks, such as summarisation, are very hard to crack by nature. This post, a follow-up to my earlier one on Wav2Vec2 trials, will outline 2 trials: Learn this step by step with the interactive AI and Data Scientist and Machine Learning roadmaps.

REPO, REQUIREMENTS, AND REFERENCES

The notebooks and audio files needed for the trials are in my repo. Additionally, you'll need these to run the notebooks:

The code in most of the notebooks have been updated to use a better approach for transcribing long audio files, via a post on Github by Hugging Face's machine learning engineer Lysandre Jik. In my earlier work-around, I used Audacity to manually split up long audio clips into smaller, more manageable pieces (audio clips longer than ~90s tend to crash local machines and Colab). Lysandre's code eliminates this step by using Librosa to stream in the long audio clip in shorter, fixed "chunks". There are several versions of the Wav2Vec2 model on Hugging Face's model hub. For this post, I'll be using the wav2vec2-large-960h-lv60-self model.

TRIAL #1: TRANSCRIBE + TRANSLATE + SENTIMENT ANALYSIS

Screen-grab on right via CNBC Televisions YouTube channel Screen-grab on right via CNBC Television's YouTube channel For this trial, I opted for US President Joe Biden's first prime time speech on Mar 11/12 2021 (depending on which time zone you are in). His speech was about 24 minutes long, and I streamed it to Wav2Vec2 via 25-second chunks. You can opt for longer or shorter chunks. I found 25s to give decent results. Details are in notebook3.0 in my repo, and I won't repeat them here. The speech is complex and Biden is known to be fighting a stutter. The transcript (download directly here) is pretty rough in some parts, especially towards the end: Download raw Wav2Vec2 transcript here Download raw Wav2Vec2 transcript here To be fair, Biden spoke pretty clearly in my view, and this is an example where the Wav2Vec2 model struggled. But the model took just 12 minutes or so on my late-2015 iMac to produce the transcript, which is way, way faster than if I were to do it manually. Next, I wanted to pipe the transcript of Biden's speech to a machine translation model and see what the quality is like for a Chinese translation. I used Hugging Face's implementation of MarianMT for this, as I've tried it on a few speeches previously. The results are borderline unusable, unfortunately. Download the full Chinese translation here: The "chain-linking" process works well from a technical perspective, and you just need about 10 additional lines of code after the Wav2Vec2 process. But the quality of the translation clearly suffers when the problematic parts of the raw transcript were fed to the MarianMT model without a thorough clean up of the missing or wrong words, and proper punctuation. I added full stops (".") at the end of each 25s-worth of text transcript from Wav2Vec2, but that clearly doesn't capture the correct start and end of each sentence in the original speech. So while it seems that "chain-linking" the NLP tasks would save a lot of time, the problems in one area can and will compound, resulting in lower quality results for tasks that are further downstream. Clearly it will take additional time to clean up the raw English transcript. But I believe doing so would improve the quality of the Chinese translation considerably. As for accurate punctuation, there is no quick way to do it within Wav2Vec2 right now. But I suspect future versions of the model would resolve this issue, seeing that a number of auto-speech-recognition services on the market already have the "add punctuation" feature. Next, I wanted to try applying sentiment analysis to Biden's speech. The combination of speech-to-text with sentiment analysis can be really useful in political or business settings, where a quick understanding of the speaker's sentiment can affect certain decisions. The raw transcript was turned into a simple data-frame, and I then used Hugging Face's transformers pipeline and Plotly to generate a "sentiment-structure" chart like the one below. I have been experimenting with these sentiment charts for a while. My earlier experiments can be found here. This worked better than the attempt at Chinese translation, though I'm sure the sentiment analysis would also benefit from a thorough clean up and the inclusion of proper punctuation in the raw English transcript. There's technically no reason why one would stop at just 3 tasks. You can easily write a few more lines of code and include summarisation as the 4th NLP task in a row. But auto-summarisation of long text documents, including speeches, remain a highly challenging task even for transformer models. The results aren't ready for prime-time, in my view. Let's take a closer look in the next trial.

TRIAL #2: TRANSCRIBE + SUMMARISE

Screen-grab via the YouTube channel of the Singapore Prime Ministers Office. Screen-grab via the YouTube channel of the Singapore Prime Minister's Office. For this I picked a shorter audio clip, a 4-minute video of Singapore Prime Minister Lee Hsien Loong responding to a question on populism at a business conference in 2019. The clip focuses on a singular issue, but has enough complexity for it to be challenging for any auto-summarisation model. See notebook3.1 in my repo for details. The Wav2Vec2 output is excellent, as you can see below (or download here). There are some minor trip ups here and there, but nothing you can't clean up really quickly. I ran the raw transcript on 2 transformer models fine-tuned for summarisation, FB's "bart-large-cnn" and Google's "pegasus-large". Here's the result from Bart: This is the summary by Pegasus: Both models did well in first capturing the speaker's broad definition of populism, followed by the inclusion of his elaboration about how Singapore tried to avoid the problem. But the FB-Bart model didn't capture any details from the second part of Mr Lee's comments. Meanwhile, the Pegasus model captured too much from the first part of his comments, and not enough of the second half. Neither version would pass for a good summary, in my view, though to be fair, both models weren't trained on political speeches. So yet again, we see that the chain-linking of NLP tasks via Wav2Vec2 and transformers is technically viable, but the results are not always satisfactory.

CONCLUSION

These are exciting times for folks working in NLP, and Wav2Vec2 looks set to open up a whole new range of possibilities. With Hugging Face initiating a sprint to extend Wav2Vec2 to other languages (beyond English), the scope for "chain-linking" NLP tasks can only grow. But given existing limitations on Wav2Vec2 and the inherent difficulties in many NLP tasks such as summarisation, it is probably wiser to add a "pause" button in the process. By that I mean results will be better if the raw Wav2Vec2 transcripts are cleaned up before they are fed downstream to other NLP tasks. Otherwise the problematic areas in the transcript get compounded, resulting in sub-par results elsewhere. As always, if you spot mistakes in this or any of my earlier posts, ping me at:

The repo for this post, containing the data and notebooks for the charts, can be found here.

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Chua Chin Hon

See all from Chua Chin Hon AIData ScienceHugging FaceMachine LearningNLP Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Getting started with PyTorch

Artificial Intelligence Learn the basics through examples André Ribeiro June 30, 20213 min read

Interpretable Machine Learning

Artificial Intelligence Extracting human understandable insights from any Machine Learning model Parul Pandey April 2, 20199 min read

Understanding Probability Models and Axioms

Artificial Intelligence Why even care about sample space, events, and probability measures? Marvin Lanhenke December 18, 20217 min read

Visualize Your Strava Data on an Interactive Map with Python

Data Science Use the Strava API with Python to download you activity data, create an elevation profile and plot your activity details on an interactive... René October 19, 20216 min read

5 Quick and Easy Tips to Become an Efficient Data Scientist in Both Python and R

Programming Having difficulty transitioning between using Python and R? Try these tips to make the transition smoother Madison Hunter November 8, 20229 min read

Implementing Streamlit-Authenticator Across Multi-Page Apps

Programming Streamlit-Authenticator allows you to add a simple yet robust method for user authentication in a Streamlit application. Brian Roepke November 23, 20246 min read

NodePiece: Tokenizing Knowledge Graphs

Machine Learning Mapping each node to an embedding vector results in enormously large embedding matrices. Is there a way to have a fixed-size vocabulary of... Michael Galkin June 24, 202110 min read

How to make your own Instagram filter with facial recognition from scratch using python

Data Science Over the past 10 years, Facial recognition technology has developed rapidly and has quickly developed a variety of uses. From the... Mitchell Krieger October 27, 20205 min read

How Metrics (and LLMs) Can Trick You: A Field Guide to Paradoxes

Data Science When numbers lie — and your metrics mislead you Subha Ganapathi July 15, 20257 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close