Video Colorization Based on a Diffusion Model Implementation

Leandro Stival*, Ricardo da Silva Torres, Helio Pedrini

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference paperAcademicpeer-review

Abstract

Cutting-edge techniques are being employed by researchers to develop algorithms that have the capability to automatically add color to black-and-white videos. This advancement has the potential to revolutionize our experience of historical films and provide filmmakers and video producers with a powerful new tool. These algorithms employ sophisticated deep neural networks to analyze images, identifying patterns and offering a promising avenue for extracting meaning and insights from visual data in the field of computer vision. Although current studies primarily focus on image colorization, there is a noticeable gap when it comes to videos and movies in the realm of deep machine learning techniques. Our investigation aims to bridge this gap and demonstrate that the image colorization techniques used today can also be effectively applied to videos and match the current state of the art presented at NTIRE 2023 video colorization challenge. We explored the application of diffusion models, which have gained popularity due to their ability to generate images and text. Our implementation involves utilizing a diffusion model to introduce noise in the frames, while a U-Net with self-attention layers predicts the denoised frames, thereby predicting the color of the video frames. For training purposes, we utilized the DAVIS and LDV datasets. When comparing the colorized frames with the ground truth in the test set, we observed promising results under several quality metrics, such as PSNR, SSIM, FID, and CDC.

Original languageEnglish
Title of host publicationIntelligent Systems and Applications
Subtitle of host publicationProceedings of the 2024 Intelligent Systems Conference IntelliSys
EditorsKohei Arai
PublisherSpringer
Pages117-131
Number of pages15
Volume1
ISBN (Electronic)9783031663291
ISBN (Print)9783031663284
DOIs
Publication statusPublished - 2024
EventIntelligent Systems Conference, IntelliSys 2024 - Amsterdam, Netherlands
Duration: 5 Sept 20246 Sept 2024

Publication series

NameLecture Notes in Networks and Systems
Volume1065 LNNS
ISSN (Print)2367-3370
ISSN (Electronic)2367-3389

Conference/symposium

Conference/symposiumIntelligent Systems Conference, IntelliSys 2024
Country/TerritoryNetherlands
CityAmsterdam
Period5/09/246/09/24

Keywords

  • Deep learning diffusion models
  • Evaluation metrics
  • Video colorization

Fingerprint

Dive into the research topics of 'Video Colorization Based on a Diffusion Model Implementation'. Together they form a unique fingerprint.

Cite this