This video is embedded from Youtube and here’s the original one
Want to block certain file types or restrict search engine crawlers from crawling certain parts of your website? That’s what a robots.txt file is for & through this video tutorial you will learn how to use robots.txt file like an expert. https://rankmath.com/kb/how-to-edit-r…
Timestamp:

00:00 Introduction
01:25 Several Use Cases of Robots.txt
03:03 Language Of Search Engines
03:16 User-Agent
03:44 Disallow Rule
03:57 Allow Rule
04:25 Crawl-Delay Rule
04:34 Sitemap Directory
04:44 Robots.txt syntax
05:43 The Location Of Your Robots.txt
06:20 Managing Robots.txt with Rank Math
07:04 Default Robots.txt Rules
07:31 The Structure of a Robots.txt File
08:29 Block Filters and Sorting Pages
09:44 Block Internal Search Pages
11:20 Blocking Certain Paths of Your Website
11:45 Block Certain File Types On Your Website
12:41 Block All Web Crawlers Except One
13:06 How To Debug Robots.txt Errors
14:39 Outro