Watch history on YouTube

Scrapy YouTube watch history spider. Because YouTube doesn't have a history search.

readme.md

Youtube History Scraper

A command line scraper written in Python 2.7 using scrapy and lxml. Scrapes your YouTube History into a searchable SQLite database, so you can search for something you've previously watched. Allowing search by video length, date published, video description, video title, etc. Uses pre logged in session cookies extracted from chrome.

This is just an experimental project. The database design isn't well thought out and there are not implemented methods for getting information back out of the database.

Ensure dependencies are met:

pip install scrapy lxml sqlalchemy

You'll need a session cookie from your web broweser, so the scraper can use your account to get your history.

Getting session cookies:

• Click the "editThisCookie" button, then click export. The cookies are now on you clipboard.
• Create the file "youtube_cookies.json" in this directory, then paste and save.

Alternitive (cookies) method:

• Open chrome / chromium
• Make sure your logged into youtube.com
• Open the inspector (F12)
• Open the network tab, check the "Preserve History" box
• Visit
• Click the request for "/feed/history" in the network tab
• Copy the section "Request Headers", below the title "Request Headers"
• Paste into a new file called "youtube_request_headers.txt" in this directory
• In the settings.py file comment the "COOKIES_JSON" line and uncomment the "CHROME_HEADERS_FILE" line.

Run Scrapy:

scrapy crawl yth_spider

Profit?

Or use ipython and sqlalchemy, this example outputs all videos > 7 minutes in length:

from db_api import * db = AppDatabase with db._session_scope as s: h_entries = s.query(HistoryEntry).filter(HistoryEntry.time > 7 * 60).limit(5) for he in h_entries.all: print "{t} ( ".format(t=he.title, v=he.vid)

The db_api module is just a wrapper around a sqlalchemy model and session. db._session_scope gives you a sqlalchemy session you can run queries on, as shown above.