Range header of the HTTP protocol

First, lets start by seeing how the HTTP protocol allows downloads to be resumed. If you don’t already have PuTTY, get it from here. We will use PuTTY to directly connect to a HTTP server, send it some HTTP headers and see the response.

Create a text file with some text in it (at least 20 characters long to make it easier for us to test). Upload it so it is accessible at http://localhost/file.txt . I have used the URL http://localhost/file.txt . Change the settings below according to your URL. Open PuTTY and use the following settings:

Host Name: localhost
Port: 80
Connection Type: Raw
Close window on exit: Never

Click on Open. In the window that shows up, type the following:

GET /file.txt HTTP/1.1
Host: localhost
Range: bytes=5-
Connection: Close

Make sure you hit enter twice after typing the text above. You should see something like the text below if you had used something like “abcdefghijklmnopqrstuvwxyz” as the contents of file.txt:

HTTP/1.1 206 Partial Content
Date: Fri, 06 Nov 2009 16:25:07 GMT
Server: Apache/2.2.12 (Win32) mod_ssl/2.2.12 OpenSSL/0.9.8k
Last-Modified: Fri, 06 Nov 2009 15:56:25 GMT
ETag: "4000000044a82-1a-477b5e2d8093c"
Accept-Ranges: bytes
Content-Length: 21
Content-Range: bytes 5-25/26
Connection: close
Content-Type: text/plain

fghijklmnopqrstuvwxyz

You can see that it has returned a few headers followed by the contents of the file starting from the 6th byte. Try it a few more times modifying the Range header (for example, 5-15 to get bytes 6 to 16).

Using cURL to resume downloads

Resuming a download using cURL is quite straight forward. We use the CURLOPT_RANGE option to set the range if the file we are trying to download already exists and use the “a” option to fopen to append the contents to the file. The code is below:

$url="http://localhost/file.txt";
$fileName = "file.txt";

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);

if (file_exists($fileName)) {
	$from = filesize($fileName);
	curl_setopt($ch, CURLOPT_RANGE, $from . "-");
}

$fp = fopen($fileName, "a");
if (!$fp) {
	exit;
}
curl_setopt($ch, CURLOPT_FILE, $fp);
$result = curl_exec($ch);
curl_close($ch);

fclose($fp);

Using fsockopen to resume downloads

Resuming a download with fsockopen is more involved as it requires us to send the request headers and then parse the response headers:

$url="http://localhost/file.txt";
$fileName = "file.txt";

$partialContent = false;
$finalFileSize = 0;

$urlParts = parse_url($url);
$socketHandler = fsockopen($urlParts["host"], 80, $errno, $errstr, 30);
if (!$socketHandler) {
	exit;
} else {
	$from = 0;

	if (file_exists($fileName)) {
		$from = filesize($fileName);
	}

	$out = "GET " . $urlParts["path"] . " HTTP/1.1\r\n";
	$out .= "Host: " . $urlParts["host"] . "\r\n";
	$out .= "Range: bytes=" . $from . "-\r\n";
	$out .= "Connection: Close\r\n\r\n";

	$headerFound = false;

	if (!$fileHandler = fopen($fileName, "a")) {
		exit;
	}

	fwrite($socketHandler, $out);
	while (!feof($socketHandler)) {
		if ($headerFound) {
			if ($partialContent) {
				$result = fread($socketHandler, 8192);

				if (fwrite($fileHandler, $result) === false) {
					exit;
				}
			} else {
				fclose($fileHandler);
				fclose($socketHandler);
				exit;
			}
		} else {
			$result = fgets($socketHandler, 8192);
			$result = trim($result);
			if ($result === "") {
				$headerFound = true;
			}

			if (strstr($result, "206 Partial Content")) {
				$partialContent = true;
			}

			if (preg_match("/^Content-Range: bytes (\d+)-(\d+)\/(\d+)$/", $result, $matches)) {
				$finalFileSize = intval($matches[3]);
			}
		}
	}
	fclose($fileHandler);
	fclose($socketHandler);

	clearstatcache();

	if (filesize($fileName) == $finalFileSize) {
		// success
	} else {
		exit;
	}
}

Related posts:

1. cURL wrapper class with executable and PHP extension support
2. Domain name WHOIS query using PHP
3. Javascript and CSS compression using YUI compressor, PHP and gzip

The class I came up with is below:

class Curl {

	// Set to "PHP" or the path to the curl executable (binary)
	var $curlExecutable = "PHP";

	// URL to request, request type (GET or POST), request parameters as an associative array or null for none
	var $url;
	var $requestType = "GET";
	var $parameters = null;

	// request headers as an array
	var $requestHeaders = array();

	// certification authority, return data with headers, follow redirects and verify peer if URL is https
	var $cainfo = null;
	var $outputWithHeaders = false;
	var $followLocation = false;
	var $SSLVerifyPeer = true;

	// connect to remote server timeout and timeout to download the entire page
	var $connectTimeout = 30;
	var $timeout = 10800;

	// if there was an error, errorNumber and error is populated. data contains the page downloaded
	var $error = null;
	var $errorNumber = null;
	var $data = null;

	function fetchPage() {

		if ($this->curlExecutable == "PHP") {
			$ch = curl_init();

			if ($this->cainfo) {
				curl_setopt($ch, CURLOPT_CAINFO, $this->cainfo);
			}

			if ($this->SSLVerifyPeer) {
				curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
			} else {
				curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
			}

			if ($this->requestType === "POST") {
				curl_setopt($ch, CURLOPT_POST, 1);

				if ($this->parameters) {
					if (is_array($this->parameters)) {

						$encodedParameters = "";
						$join = "";

						foreach ($this->parameters as $name => $value) {
							$encodedParameters = $encodedParameters . $join . rawurlencode($name) . "=" . rawurlencode($value);
							$join = "&";
						}

						curl_setopt($ch, CURLOPT_POSTFIELDS, $encodedParameters);

					} elseif (is_string($this->parameters)) {

						curl_setopt($ch, CURLOPT_POSTFIELDS, $this->parameters);

					}
				}

			} else {
				curl_setopt($ch, CURLOPT_POST, 0);

				if ($this->parameters) {
					$encodedParameters = "";
					$join = "?";

					foreach ($this->parameters as $name => $value) {
						$encodedParameters = $encodedParameters . $join . rawurlencode($name) . "=" . rawurlencode($value);
						$join = "&";
					}

					$this->url = $this->url . $encodedParameters;
				}

			}

			curl_setopt($ch, CURLOPT_HTTPHEADER, $this->requestHeaders);

			if ($this->outputWithHeaders) {
				curl_setopt($ch, CURLOPT_HEADER, 1);
			} else {
				curl_setopt($ch, CURLOPT_HEADER, 0);
			}

			if ($this->followLocation) {
				curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
			} else {
				curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
			}

			curl_setopt($ch, CURLOPT_URL, $this->url);
			curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

			curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $this->connectTimeout);
			curl_setopt($ch, CURLOPT_TIMEOUT, $this->timeout);

			$this->data = curl_exec($ch);
			$this->error = curl_error($ch);
			$this->errorNumber = curl_errno($ch);
			curl_close($ch);

			if ($this->error) {
				return false;
			} else {
				return true;
			}

		} else {

			$command = $this->executable;

			$command = $command . " -s -S";

			$command = $command . " --connect-timeout " . $this->connectTimeout;
			$command = $command . " --max-time " . $this->timeout;

			if ($this->cainfo) {
				$command = $command . " --cacert \"" . $this->cainfo . "\"";
			}

			if ($this->outputWithHeaders) {
				$command = $command . " -i";
			}

			if ($this->followLocation) {
				$command = $command . " -L";
			}

			for ($i = 0; $i < count($this->requestHeaders); $i++) {
				$command = $command . " -H \"" . $this->requestHeaders[$i] . "\"";
			}

			if ($this->parameters){
				$encodedParameters = "";
				$join = "";

				foreach ($this->parameters as $name => $value) {
					$encodedParameters = $encodedParameters . $join . rawurlencode($name) . "=" . rawurlencode($value);
					$join = "&";
				}

				$command = $command . " -d " . escapeshellarg($encodedParameters) . "";
			}

			if ($this->requestType === "GET") {
				$command = $command . " -G";
			}

			$command = $command . " " . $this->url;

			$descriptorspec = array(
			0 => array("pipe", "r"),
			1 => array("pipe", "w"),
			2 => array("pipe", "w")
			);

			$process = proc_open($command, $descriptorspec, $pipes);

			if (is_resource($process)) {

				fclose($pipes[0]);    // close stdin as we are not going to supply any input

				$this->data = stream_get_contents($pipes[1]);    // read stdout
				$this->error = stream_get_contents($pipes[2]);    // read stderr

				fclose($pipes[1]);
				fclose($pipes[2]);

				$this->errorNumber = proc_close($process);

			} else {

				return false;

			}

			if ($this->errorNumber === 0) {
				return true;
			} else {
				return false;
			}

		}

	}

}

The following example fetches http://www.google.com :

$request = new Curl;
$request->curlExecutable = "PHP";
$request->followLocation = true;
$request->url = "http://www.google.com";
if ( $request->fetchPage() ) {
	header("Content-Type:text/plain");
	echo $request->data;
} else {
	echo "An error occured: " . $request->errorNumber . " - " . $request->error;
}

This example fetches a random quote (the ones tou see on the top right of this website):

$request = new Curl;
$request->curlExecutable = "PHP";
$request->url = "http://www.ankur.com/quotes.php";
$request->requestType = "GET";
$request->parameters = array(
	"action" => "random"
);
if ( $request->fetchPage() ) {
	header("Content-Type:text/plain");
	echo $request->data;
} else {
	echo "An error occured: " . $request->errorNumber . " - " . $request->error;
}

Unrelated to the cURL wrapper class, you can try changing the action parameter from random to all to fetch all the quotes.

Related posts:

1. AES wrapper class with pure php and MCrypt extension support
2. Resume HTTP downloads in PHP using cURL or fsockopen