Building Video Sharing Platform

System Requirements and Objectives

For this exercise, we aim to design a streamlined version of a video-sharing platform with the following features:

Functional Objectives:

1. Users can upload video content.
2. Videos can be shared and watched by other users.
3. Search functionality to locate videos based on their titles.
4. The system should track video statistics, including likes, dislikes, and view counts.
5. Users can leave and read comments on videos.
6. The platform must ensure high reliability, preventing any loss of uploaded videos.
7. The service should prioritize availability. Occasional delays in video visibility are acceptable if it ensures high availability.
8. Users should enjoy a seamless video-watching experience without noticeable lag.

Capacity Estimation and Constraints

Let’s assume there are 2 billion users in total, with 900 million active daily users. If an average user views six videos per day, the total number of video views per second would be:

900M * 6 / 86400 sec = 62.5K video views/sec

Assuming an upload:view ratio of 1:250, meaning for every 250 views, one video is uploaded, the upload rate would be approximately:

62.5K / 250 = 250 videos/sec

Storage Requirements: Assume 600 hours of videos are uploaded every minute. If one minute of video requires 60MB for storage (including multiple formats), the total storage needed per minute would be:

600 hours * 60 minutes * 60MB = 2160 GB/min (36 GB/sec)

These figures exclude compression and data replication, which would affect storage needs.

Bandwidth Requirements: With 600 hours of video uploaded per minute and each minute requiring 15MB of bandwidth, uploads would consume:

600 hours * 60 minutes * 15MB = 540GB/min (9GB/sec)

Given an upload:view ratio of 1:250, the outgoing bandwidth requirement would be roughly:

9GB * 250 = 2.25TB/sec

API

We can use RESTful APIs to enable the core functionalities of our platform. Below are examples of API definitions for video uploads and searches:

uploadVideo(api_key, title, description, tags[], category_id, language, video_metadata, video_data)


Parameters:

• api_key (string): A unique key for authenticated API access.
• title (string): The title of the video.
• description (string): Optional description of the video.
• tags (array): Optional tags to categorize the video.
• category_id (string): A category such as Music, Gaming, or Education.
• language (string): The default language, e.g., English, Spanish, etc.
• video_metadata (string): Location details of where the video was created.
• video_data (stream): The actual video file to be uploaded.

Response:

• Returns HTTP 202 (Accepted) for successful uploads. The user is notified via email upon encoding completion with a link to the video. Additionally, an API can provide status updates for uploads.

searchVideo(api_key, query, max_results, pagination_token)


Parameters:

• api_key (string): Authenticated API access key.
• query (string): Search terms.
• max_results (integer): The maximum number of results per query.
• pagination_cursor (string): Token to fetch additional pages of results.

Response:

• JSON object with video details including title, thumbnail URL, upload date, and view count.

Chunked Uploads and Presigned URLs

Handling large video uploads efficiently is a critical aspect of the platform. To improve resilience and minimize the impact of network interruptions, the system supports chunked uploads and presigned URLs.

Chunked Uploads:

Chunked uploads allow users to upload large video files in smaller segments. This approach ensures that if the upload process is interrupted, only the incomplete chunks need to be retransmitted. The platform tracks the progress of each chunk using a unique session ID associated with the upload process. Key advantages include:

1. Resilience: Minimizes the need for users to restart large uploads from scratch.
2. Efficiency: Reduces bandwidth wastage caused by failed uploads.
3. Scalability: Enables parallel uploads of multiple chunks to optimize upload speeds.

Workflow:

1. The client initiates the upload session, receiving a unique session ID.
2. Each chunk is uploaded sequentially or in parallel, with metadata specifying the chunk number and session ID.
3. The server assembles the chunks upon completion and verifies the file integrity using checksums.

Presigned URLs:

Presigned URLs enable clients to upload video files directly to the storage service (e.g., Amazon S3) without routing the file through application servers. This reduces server load and improves upload performance.

Workflow:

1. The client requests a presigned URL from the platform’s API for the file or chunk to be uploaded.
2. The platform generates a URL with an expiration time and permissions specific to the upload.
3. The client uploads the file or chunk directly to the storage service using the presigned URL.
4. Upon successful upload, the storage service notifies the platform, which updates the upload status.

Benefits:

1. Reduced Load: Application servers are bypassed for file data, handling only metadata.
2. Security: URLs are time-limited and restricted to specific actions.
3. Performance: Direct uploads improve speed and reduce latency.

High-Level Architecture

The system architecture would include the following components:

1. Task Queue: Each uploaded video is placed in a queue for video processing tasks like encoding.
2. Encoding Service: Converts uploaded videos into multiple formats to support diverse devices.
3. Storage Database: Distributed file storage for videos.
4. User Database: Stores user details such as name, email, and preferences.
5. Metadata Database: Contains metadata about videos, such as titles, storage locations, uploader information, view stats, likes/dislikes, and comments.

Database Schema

1. User Table

2. Video Table

3. Comment Table

4. Video Tag Table

5. Metadata Table

Detailed Component Design

The service would primarily handle a read-heavy workload, necessitating optimization for fast video retrieval. Assuming a read:write ratio of 250:1, efficient read management is critical.

Video Storage: Videos will be stored in distributed systems such as Amazon S3, which support scalability and redundancy.

Managing Read Traffic: To handle massive read volumes, traffic should be separated between read and write operations.

After video upload to Blob Storage, the media will be distributed and served through a CDN allowing handling ready heavy traffic. CDNs replicate videos across globally distributed edge servers, enabling content delivery from locations closer to the end-users. This significantly reduces latency and improves streaming performance.

Techniques to Enhance CDN Efficiency:

1. Cache Warm-Up:Before a video is expected to gain high traffic (e.g., after a premiere or major upload), the system proactively pushes video content to CDN edge servers, ensuring faster delivery when users request it.
2. Regional Popularity-Based Caching:Videos are cached in regions where they are trending. For example, a video gaining traction in North America will have more CDN replicas deployed in that region.
3. Multi-Resolution Caching:CDNs store multiple resolutions (e.g., 720p, 1080p, 4K) to cater to devices with varying capabilities and bandwidth conditions, allowing adaptive streaming without excessive buffering.
4. Load-Based Scaling:The CDN dynamically allocates resources based on traffic patterns, scaling up for popular content and reallocating resources as traffic subsides.
5. Efficient Cache Invalidation:When videos are updated (e.g., corrected subtitles or changes to the file), the system ensures outdated copies in the CDN are invalidated promptly to prevent users from accessing stale content.

For metadata, a master-slave architecture allows writes to the master while slaves handle reads, accepting temporary staleness in exchange for faster access when fetching from a distributed cluster of servers serving read heavy requests.

Video Upload Resilience: As video files are large, uploads should support resumable transfers in case of network interruptions. See video upload in chunks and pre-signed urls above.

Video Encoding: Uploaded videos will be queued for encoding into various formats. Once processing is complete, the user will be notified, and the video will be available for viewing and sharing.

Metadata Sharding

To handle the massive data volume and high read load efficiently, we need to distribute metadata across multiple servers. Here are different sharding strategies:

Sharding by UserID:

• All metadata for a specific user is stored on a single server, determined by hashing the UserID. Queries for a user's data are directed to the appropriate server.
• Drawbacks:
  1. Popular users could overwhelm their assigned server, creating a bottleneck.
  2. Uneven data growth among users makes maintaining a balanced load challenging.

Sharding by VideoID:

• Each video’s metadata is stored on a server determined by hashing its VideoID. This prevents single-user bottlenecks but may lead to issues with highly popular videos.

Optimizations:

• Use consistent hashing to balance load dynamically and minimize data redistribution when adding/removing servers.
• Introduce a caching layer to store frequently accessed metadata for faster retrieval.

Load Balancing

To manage server load effectively:

• Use consistent hashing to distribute requests evenly among cache servers.
• Implement dynamic HTTP redirections to shift traffic from overloaded servers to less busy ones.

Cache

Caching strategies include:

• Use geographically distributed caches for faster access.
• Implement LRU (Least Recently Used) eviction to prioritize active data.
• Cache popular metadata and videos to reduce database load.

‍