Design:MirrorIt

From RecodedWiki

This page is a branch from MirrorIt

Project Overview

Plan

Goals, objectives and target audience

MirrorIt will provide users and developers with a simple way to get multiple mirrors of their software or other media with little effort. Users are able to search for files they want to find and download them with ease from a server selected mirror.

Look and feel

The web server will provide a simplistic interface with ease of use in mind, allowing users and developers to quickly find what they want. From the main page, users and developers will easily find powerful search features, a short list of the latest mirrors added and a link to the page for setting up new mirrors.

All major web browsers will be supported to have the best compatibility between them and the site. This includes popular browsers for Windows, Linux and Mac. Also included is support for text, mobile phone and text to speech based browsers. The minimum screen resolution of the web site will be 800 x 640.

Hardware and software requirements

The MirrorIt web server will need to be fast in responding to web page requests including multiple sql queries through PHP. The MirrorIt Premium Mirror Servers will need to have fast data transfers and large amounts of storage space.

Each server will run Debian Linux and Apache (with PHP and MySQL on the web server)

Project management

Each server will need a way for maintenance and upgrades to be done, as well as troubleshooting. For the main local server or anything that can be done remotely this will not be a problem, however after adding servers in other countries we will need to hire people for these tasks.

Building the site

Discovery

Content

The content of ALL pages will include:

Site banner vertically aligned to top, horizontally aligned to center.
Tabs below site banner to select category, Login/Register or Add New Mirror pages.
Search bar located immediately below category tabs.
Page specific content below search bar. Advertisements on the side of the page.
Footer at bottom of page with links for copyright, policy and FAQ.

The content of the MAIN page will show a summarization of:

The top 10 rated files...
- By number of mirrors.
- By number of downloads.
The last 10 newly added files.

The content of pages by category will contain:

Options for sorting by mirrors, downloads, date, size, etc..
List of files in requested sort order (default by date, newest first)

The Add New Mirror page will contain a form for setting up a mirror for a new file to be added to the database. If the user is not logged in, it will redirect them to the Login/Register page.

The Login/Register page will have a form to allow users to either Login or Register a new account.

Risk assessment

[ToDo: Assess the risks that might need special attention. Determine how to handle those risks.]

Design

Conceptual design and prototypes

Check mirrorit.recoded.net[1] for the latest mockup.

Technology architecture

PHP will be used for the front end.

MySQL databases will be used for mirror information as well as user accounts.

GeoIP (or similar) location tracker will be used to help select the fastest mirrors for downloading based on user location and mirror locations.

PayPal will be used for purchasing credits.

Captcha will be used for 'human verification' on creating accounts and adding new files. Captcha isn't exactly bot proof (I can demonstrate this easily with ANY Captcha) but it will stop the majority of bot submissions.

Implementation

Content development

Main page

User selectable themes done in CSS

Category tabs & pages

Search results page

Login/Register page

New submissions page with:

New File form/page
New mirror form/page

FAQs

Policy's

Stabilization

Deployment

We should target developers (of freeware/shareware that they typically struggle to find mirrors for) and perhaps the file sharing community at large (via forums).

Maintenance plan

Most maintenance will be done with remote login when possible.

In the future, servers will be set up in other locations including other countries in order to provide better speeds for users outside the US. These servers will at times require maintenance done on them that could not be done with remote access. We will need to hire a tech for these servers.

Site Functionality

Browsing

Example user session browsing for a file.: Users selects a category and/or search string/pattern.; Server responds with list of matching results based on the query issued using search pattern. Results are displayed with limited important information, such as a filename, short description, size, date added and ranking.; User clicks on one of the file links.; Server responds with page showing FULL details of the file, download link, MirrorIt link and onsite media player if the file is audio/video of a supported format.; User views/listens with media player, downloads, chooses to add a mirror or goes back to search results.

Viewing And Downloading

Playing audio/video: Supported file types for audio and video can be streamed directly onto the page via a flash media player that will automatically be placed on a page if it is known to be a supported file type. Playback of the media in the flash player follows much of the same process as downloading does.

Downloading files

Regular download...

User clicks the non-premium download button.
(Optional) Users gets users location using a Geo Location service.
Determine if user is requesting 'download resume'.
Server selects optimal user hosted mirror, based on location (if using location tracking) and last known good speeds (which is effected by number of users currently download from that mirror).
Server begins process of downloading from mirror and redirecting to client in a proxy/tunnel like fashion. This protects credentials used by users providing the mirror (See #Security).
Server verifies the download is going smoothly. If it fails, go back to mirror selection and select the next optional mirror. Resume from last known point if possible. Mark failed mirror as temporarily broken so it can be reinspected later.

Premium download...

User clicks the premium download button.
Server checks if user is logged in.
- If false, redirect to Login/Register page. Keep ID of download page in url for fallback after login/registration.
- Else verify user has enough credits for the download. Redirect to account page if false. Again, keep ID of download page for fallback when done.
From this point on, everything is the same as regular downloads except for the mirror servers used to select from. The majority of these mirrors will be MirrorIt servers specifically designed for high speed downloading and file hosting.

Mirrors

Adding a new file to be mirrored

Any user with a (Free) MirrorIt account can create a new listing for a file they want mirrored so long as they meet the following criteria:

The user must supply a valid working direct link for the original file download.
The original hosting server for the file must be of a supported protocol with valid credentials for login if necessary.
The hosting server must support download resume.
The user must accept the Terms & Conditions.
The user must show he is 'hopefully' human by completing a Captcha.

At this point, the first download from the users mirror is transfered to a MirrorIt server for premium users.

Adding a new mirror for a preexisting file.: Any logged in user may provide an additional mirror for a file that already exists in the database.; The process is the same as starting a new mirror, with one additional step mentioned below.; The user will need to provide the ID for the file they are mirroring.; After submitting the form, the server will verify that the mirror is working and is an unmodified version of the original by downloading a small portion of the file from a randomly selected start point and comparing a hash. This method reduces server load on the new mirror by not having to download the full file.

Both: Testing and information gathering on the file and host server are obtained for analytics used to make the site faster and better for mirror selection.

Mirror Verification: Verifying a mirror is successful doesn't just happen when a user first adds themselves as a mirror or creates a new mirror. This verification procedure needs to occur a bit more frequently in order to ensure that the file doesn't get changed. It also needs to be sure that the file wasn't removed and that the mirror's host is still active.

We do this to prevent broken and/or misleading downloads. Imagine a user trying to download the hottest new freeware game to hit the internet, only to end up with something completely different or malicious. (I do cover malicious software in the #Security section)

The MirrorIt server will periodically go through it's database to ensure that the host is still up, the file still exists and that it is indeed still the correct file.

Inspection occurs more often for popular files, not to exceed every 4 hours. Less frequently for less popular files, but still at least every 24 hours.
As with a new mirror, the MirrorIt server will keep the server load of the mirror to a small amount by requesting a download resume of the file from a randomly selected location and compare the small portion of the file it downloads to a hash from the database for that same portion.
If a file is modified in a section just outside of the hash check, it will be discovered with an inspection at a later point, as the chunk downloaded is randomly selected at every inspection period.

Files that fail an inspection are dealt with accordingly:

Failed hash check, mirror for this file is removed from the database immediately and the user is sent a notification explaining why so they may fix the error.
Download fails, the mirror is marked as broken (in the database) which removes it from the listings if this is the only mirror for it. The file is still tested periodically as scheduled. After two days of failure, the user is notified to fix the problem. After one week of failure, the mirror is removed from the database.

User Persuasion, Getting Mirrored

Quick overview: MirrorIt needs files to be mirrored by users in order to become a big name download service so that users want to use it.; In order to get users to want to provide a mirror after they have already obtained the file they want anyway, we need to persuade them. Sad but true.

MirrorIt Credits: Credits will be used for premium downloads. Users can purchase credits or they can obtain credits for free by participating in creating mirrors for files.; Creating mirrors for a select number of files will provide the user with 1 free MirrorIt Credit.; Currently a 5:1 ratio of mirrors:credits is suggested. Mirror 5 files, get one premium download credit.

Security

Malicious Files

What?: Malicious files are viruses, trojans, malware and generally anything that would cause havoc on a users computer or operating system.

Why?: If these were to become common place on the site, users would be discouraged to use the site any longer.

Risk?: Generally, these would be low risk, as users will not want to mirror something that isn't what they thought it was. However, sometimes these malicious files are actually disguised as something the user did want without them even noticing.

Prevention?: All new files will be scanned for malicious content by the MirrorIt file host servers during the initial backup of the original mirror.; Files that are a positive match for known virii will not be acccepted.

False positives?: If a file is scanned and returns a result that signifies it may be infected, but is not critical, it will still be placed on the site.; Any file that is suspected of containing malicious code, but is not defined as critical or 100% positive, it will be marked with it's AV scan statistics on it's details page along with a warning to users.

Mirror Server Credentials

What?: Some hosts may require a username and password in order to download from it.; They may also be on a non-default port as a means to make it private, but viewable by known friends/associates without a login.

Why?: Users may want to create mirrors but do not want to open up their HTTP/FTP server to the public.; They would want to require a login. This login would contain information that should not be made public.; Users don't want their private server (on a non-default port) from getting public attention.; A user may not want their specific hostname/IP getting increased attention from the public, even if it is a public site already.

Handling it?: The MirrorIt server acts as a proxy/tunnel to all mirrors. Because of this, only the MirrorIt server needs to know the specifics on the server for obtaining the file that is mirrored.; This means users downloading the file will never see the IP, Domain name, port or login credentials required by the server.