Add Stunning AI B-Roll toVideos for Free: A Complete Tutorial | by Anil Chandra Naidu Matcha | Jul, 2024

In in the present day’s information, I’ll present you easy methods to seamlessly use AI know-how to rapidly discover and incorporate the right B-roll, saving you time and enhancing your movies effortlessly. First let’s perceive what’s a B-roll.

B-roll refers to supplementary footage that’s intercut with the primary footage in a video. It enhances the storytelling by offering further context, supporting the narrative, or illustrating factors being made within the main footage (often known as A-roll). B-roll can embrace quite a lot of clips, equivalent to cutaways, transitions, or background pictures, that assist to create a extra partaking and visually interesting video. For content material creators, efficient use of B-roll can considerably elevate the manufacturing high quality, making movies extra dynamic and fascinating for the viewers.

The method of figuring out B-roll for a video includes both taking pictures them individually or selecting related footage from on-line sources equivalent to inventory suppliers like Pexels and so forth. manually. So let’s now try to automate this course of

This AI B-Roll generator is an open-source software out there on GitHub. It makes use of AI to automate the method of discovering and integrating B-roll footage into your movies. Beneath is the workflow we observe so as to add b-roll to our video

1. Generate captions for enter video
2. Establish key phrases which symbolize these captions

• Instance : AI is used to automate most of the human duties -> Automation

3. Fetch pexels movies for these key phrases to make use of as b-roll movies

4. Sew collectively the b-roll movies with unique video

Let’s do these step-by-step

First, you’ll want to put in the required dependencies. These embrace:

1. Whisper Library: Used to generate captions to your audio.
2. Pytube: Used to obtain movies from YouTube.
3. OpenAI: Used for AI-related duties, together with key phrase technology.
4. MoviePy: Used for video enhancing.

We begin by downloading the enter video utilizing Pytube. You may both enter a YouTube URL or add a video on to Google Colab. We will modify video_url from the beneath code to obtain any video of our alternative. The video shall be obtain to file named video.mp4

from pytube import YouTube
import os
# Operate to obtain a YouTube video
def download_youtube_video(url, output_path='.', filename=None):
attempt:
# Create a YouTube object
yt = YouTube(url)
# Get the best decision stream out there
video_stream = yt.streams.get_highest_resolution()
# Obtain the video with the desired filename
downloaded_file_path = video_stream.obtain(output_path=output_path, filename=filename)
# If a filename is specified, rename the file
if filename:
base, ext = os.path.splitext(downloaded_file_path)
new_file_path = os.path.be a part of(output_path, f"{filename}")
os.rename(downloaded_file_path, new_file_path)
print(f"Downloaded and renamed to: {new_file_path}")
else:
print(f"Downloaded: {downloaded_file_path}")
besides Exception as e:
print(f"An error occurred: {e}")
# Instance utilization
if __name__ == "__main__":
# URL of the YouTube video to be downloaded
video_url = 'https://www.youtube.com/watch?v=8ZyShHwF_g0'
# Output path the place the video shall be saved
output_path = '.'
# Obtain the video
download_youtube_video(video_url, output_path, filename="video.mp4")

Subsequent, we extract the audio from the video utilizing FFmpeg and move it to the Whisper library to generate captions utilizing the beneath command. If we’re operating the code in Google colab, ffmpeg is put in by default and therefore there is no such thing as a want to put in it individually. We then load whisper medium and run by way of our audio to generate the captions

!ffmpeg -i video.mp4 -ab 160k -ac 2 -ar 44100 -vn audio.wav
import whisper
# Load the mannequin
mannequin = whisper.load_model("medium")
outcome = mannequin.transcribe("audio.wav")

As soon as now we have the captions, we divide them into teams of 20 sentences. We do that as we noticed that if now we have loads sentences the key phrases usually are not recognized for each sentence. As soon as the division is full we move it to the OpenAI API to generate related key phrases for every sentence in a bunch. These key phrases symbolize the primary concepts within the captions and are used to seek out matching B-roll footage.

segments = outcome["segments"]
extracted_data = [{'start': item['start'], 'finish': merchandise['end'], 'textual content': merchandise['text']} for merchandise in segments]
knowledge = [x["text"] for x in extracted_data]
def split_array(arr, max_size=20):
# Record to retailer the cut up arrays
outcome = []
# Iterate over the array in chunks of dimension max_size
for i in vary(0, len(arr), max_size):
outcome.append(arr[i:i + max_size])
return outcome
# Instance utilization
my_array = record(vary(100))  # Instance array with 100 components
split_arrays = split_array(knowledge, max_size=20)

Now every of those group of 20 sentences is handed to a immediate as may be seen within the code beneath to determine visible key phrases which can be utilized to fetch related pexels movies. Lastly we merge the key phrases from every group to get the entire broll knowledge which comprises key phrases for every sentence of enter video.

from openai import OpenAI
import json
OPENAI_API_KEY = "openai-api-key"
broll_info = []
os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY
shopper = OpenAI(
api_key=OPENAI_API_KEY,
)
for i, x in enumerate(split_arrays):
immediate = """This can be a transcript from a shorts video with 20 sublists. Every sublist represents a section of the dialog. Your activity is to determine a key phrase from every sublist that can be utilized to seek for related b-roll footage. B-roll footage ought to complement the dialog matters and ought to be recognized such that it may give related outcomes when searched in pexels api. Please present one key phrase per sublist. By no means skip any sublist and at all times give so as i.e from 0 to 19. Want output with key phrase and record index. Strictly give jsonnn**Enter**nn"""+str(x)+"""nn**Output format**nn[{"k": keyword1, "i":0},{"k":keyword2, "i":1}]"""
chat_completion = shopper.chat.completions.create(
messages=[
{
"role": "user",
"content": prompt,
}
],
mannequin="gpt-4o",
)
broll_data = chat_completion.selections[0].message.content material
print("Knowledge", broll_data)
attempt:
broll_data = json.masses(broll_data)
besides:
broll_data = broll_data.cut up('```json')[1].cut up('```')[0].exchange('n', '')
broll_data = json.masses(broll_data)
broll_data = [{"k":x["k"], "i":20*i+x["i"]} for x in broll_data]
broll_info.lengthen(broll_data)

Now that now we have key phrase for every sentence of the dialog in a video, we are going to choose 50% of the key phrases for fetching B-roll movies. This manner 50% of the unique video shall be coated with b-rolls. Based mostly on requirement you may select the next or decrease proportion of b-rolls.

import random
num_to_select = int(len(broll_info) * 0.5)
enumerated_list = record(enumerate(broll_info))
selected_with_indices = random.pattern(enumerated_list, num_to_select)
selected_elements = [elem for index, elem in selected_with_indices]
selected_indices = [index for index, elem in selected_with_indices]
for x in selected_indices:
ingredient = broll_info[x]
extracted_data[x]["video"] = fetch_pexels_video(ingredient["k"])

With the key phrases in hand, we use the Pexels API to fetch related movies. First you have to create an account and Pexels and from documentation want to repeat the Pexels api key and paste within the beneath code. If we don’t discover any movies for a key phrase we return “Invalid key phrase”

import requests
PEXELS_API_KEY = "pexels-api-key"
def fetch_pexels_video(key phrase, orientation="panorama"):
url = f"https://api.pexels.com/movies/search?question={key phrase}&orientation={orientation}&dimension=medium"
headers = {
"Authorization": PEXELS_API_KEY
}
response = requests.get(url, headers=headers)
knowledge = response.json()
if knowledge['total_results'] > 0:
video_info = knowledge['videos'][0]
video_url = video_info['video_files'][0]['link']
thumbnail_url = video_info['image']
video_url = knowledge['videos'][0]['video_files'][0]['link']
return {'video': video_url, 'thumbnail': thumbnail_url}
else:
return "Invalid key phrase"

Now that now we have all of the content material available, we are able to sew every little thing collectively. We iterate over all of the sentences within the captions, creating clips for every sentence. If a sentence is chosen for a B-roll, we incorporate the corresponding B-roll video; in any other case, we retain the unique clip. The B-roll clips are resized and trimmed to match the length of the respective sentences.

Now we add the audio again to the created video above utilizing concatenate_clips_with_audio operate to sync each and generate the ultimate video.

import os
import requests
from moviepy.editor import VideoFileClip, concatenate_videoclips, concatenate_audioclips
from tempfile import TemporaryDirectory
from moviepy.video.fx.all import resize
def download_video(url, temp_dir):
local_filename = os.path.be a part of(temp_dir, url.cut up('/')[-1])
with requests.get(url, stream=True) as r:
r.raise_for_status()
with open(local_filename, 'wb') as f:
for chunk in r.iter_content(chunk_size=8192):
f.write(chunk)
return local_filename
def process_broll_clip(b_roll_clip, segment_duration, original_audio, begin):
b_roll_duration = b_roll_clip.length
if b_roll_duration < segment_duration:
num_loops = int(segment_duration / b_roll_duration) + 1
b_roll_clip = concatenate_videoclips([b_roll_clip] * num_loops)
b_roll_clip = b_roll_clip.subclip(0, segment_duration)
else:
b_roll_clip = b_roll_clip.subclip(0, segment_duration)
b_roll_clip = resize(b_roll_clip, newsize=(original_clip.w, original_clip.h))
# Set audio from the unique video to the b-roll clip
b_roll_clip = b_roll_clip.set_audio(original_audio.subclip(begin, begin + segment_duration))
return b_roll_clip
def concatenate_clips_with_audio(clips):
audio_clips = [clip.audio for clip in clips if clip.audio is not None]
video_clips = [clip for clip in clips]
final_video = concatenate_videoclips(video_clips, technique="compose")
if audio_clips:
final_audio = concatenate_audioclips(audio_clips)
final_video = final_video.set_audio(final_audio)
return final_video
# Load the unique video
original_video_path = 'video.mp4'
original_video = VideoFileClip(original_video_path)
original_audio = original_video.audio
with TemporaryDirectory() as temp_dir:
final_clips = []
for section in extracted_data:
begin = section['start']
finish = section['end']
segment_duration = finish - begin
original_clip = original_video.subclip(begin, finish)
if 'video' in section and section["video"] != "Invalid key phrase":
print("Phase", section)
b_roll_video_url = section['video']['video']
b_roll_video_path = download_video(b_roll_video_url, temp_dir)
b_roll_clip = VideoFileClip(b_roll_video_path)
b_roll_clip = process_broll_clip(b_roll_clip, segment_duration, original_audio, begin)
final_clips.append(b_roll_clip)
else:
final_clips.append(original_clip)
final_video = concatenate_clips_with_audio(final_clips)
final_video.write_videofile('final_video_with_broll.mp4', audio_codec='aac')

Beneath is a demo video with AI b-roll added. Right here is the unique video https://github.com/Anil-matcha/AI-B-roll/blob/main/video.mp4