How to Use cURL for Sending GET Requests

Master web scraping with our quick guide on cURL for GET requests, unlocking the power of data transfer and API integration effortlessly.

Profile picture of Zawwad Ul Sami
Zawwad Ul Sami
Cover Image for How to Use cURL for Sending GET Requests

cURL serves as a command-line tool and library designed for efficiently transferring data between servers and your system through diverse protocols. In addition to retrieving public data, such as HTML, text, JSON, and images, cURL is proficient in managing sessions and cookies. Its capability to interface with APIs and seamless integration with shell commands makes it an excellent choice for automation. These versatile features collectively position cURL as a powerful tool, particularly well-suited for web scraping tasks.

Leveraging a cURL GET request proves instrumental in retrieving public data. In this article, we will understand the process of executing GET requests for data extraction.

What does a GET request mean?

A GET request in the context of HTTP involves soliciting data from a designated web page. For instance, when you input a URL into your browser and initiate the action, your browser dispatches a GET request to the hosting server, seeking to obtain the corresponding HTML code.

Although various HTTP request methods can lead to a web server providing you with HTML, these methods serve distinct purposes. Take HTTP POST, for instance, designed specifically for submitting data. When you click the Login button, it utilizes HTTP POST to dispatch your username and password to the server. Even though you still view a web page, the POST request method is primarily for transmitting form data. If you're interested in mastering the art of sending POST requests, explore our dedicated blog post on POST requests with cURL.

On the other hand, a GET request doesn't send any information. It's solely for asking for a webpage or other resources, such as images. Let's look at a straightforward example of a cURL GET request:

curl http://example.com

The outcome is the HTML provided by the web server.

How to Use cURL for Sending GET Requests: A Step-by-Step Guide

Now that we have grasped the fundamentals, let's explore the process of sending a GET request with cURL. This tutorial will utilize httpbin.org, a straightforward HTTP request and response service. Our exploration of cURL will encompass various aspects, including:

  • Straightforward cURL GET Requests
  • Issuing a GET Request with Parameters
  • Fetching GET HTTP Headers
  • Receiving Responses in JSON Format
  • Handling Redirects
  • Including Cookies with a GET Request

First Step - Basic cURL GET Requests

If the default request method is already set to GET, you can bypass the --request option and initiate a GET request like this:

bashCopy codecurl http://example.com

Second Step - Executing a GET Request with Parameters

A GET request with parameters enables you to include extra data in your request's URL and send it to the server. cURL offers two robust options, -d and -G, to simplify this process.

It's important to note that using -d without -G will result in a POST request. Additionally, employing the -X option with the GET value will cause the data intended for -d to be disregarded.

Consequently, to successfully send parameters with a GET request, it's imperative to use -G in conjunction with -d.

In the provided command, 'param1' and 'param2' serve as keys, with 'value1' and 'value2' representing their respective values. The -d option can be employed multiple times to transmit different parameters.

Alternatively, the GET parameters can be integrated directly into the URL:

bashCopy codecurl http://example.com?param1=value1&param2=value2

Third Step - Retrieving GET HTTP Headers

HTTP headers facilitate the exchange of extra information between the client and server during an HTTP request. To obtain both the HTTP headers and the response body, incorporate the -i or --include option in the cURL GET command:

bashCopy codecurl -i http://example.com

This command fetches the HTTP response headers, including details like the server, date, content type, and content length. These headers offer valuable insights into the characteristics and specifics of the response data.

It's worth noting that the extended parameter for retrieving response headers is --head:

bashCopy codecurl --head http://example.com

Fourth Step - Obtaining Data in JSON Format

JSON has emerged as a standard for data exchange in today's web development landscape. When dealing with APIs, it's essential to request data in JSON format. You can command cURL to expect the response in JSON format by utilizing the -H option followed by "Accept: application/json":

bashCopy codecurl -H "Accept: application/json" http://example.com

Please be aware that including "Accept: application/json" doesn't ensure that the response will be provided in JSON format. The outcome largely relies on whether the website supports delivering a response in JSON.

Fifth Step - Tracking Redirects

In specific situations, the URL you're trying to access might redirect to another URL. By default, cURL doesn't automatically follow these redirects, but you can explicitly command it to do so. You can accomplish this by employing the -L or --location option:

bashCopy codecurl -L http://example.com

Sixth Step - Incorporating Cookies in a GET Request

At times, it becomes necessary to include cookies in your GET request, particularly when dealing with websites that rely on user sessions or track user activity. To achieve this, employ the -b or --cookie option, followed by the cookie's name and value:

bashCopy codecurl -b "sessionid=12345" http://example.com

Parameters for cURL GET Requests

To understand the process of sending GET requests with cURL, we've compiled a table highlighting essential arguments, including the short option, the long option, and their respective descriptions.

| Argument         | Short Option | Long Option   | Description                                          |
|------------------|--------------|---------------|------------------------------------------------------|
| Headers          | -I           | --head        | Retrieves HTTP headers only.                        |
| Include Headers  | -i           | --include     | Includes the HTTP response headers in the output.    |
| User Agent       | -A           | --user-agent  | Specifies the User-Agent string to send to the server. |
| Request Type     | -X           | --request     | Specifies the request type to use (GET, POST, PUT, DELETE, etc.) |
| Follow Redirects | -L           | --location    | Follows redirects in the server's response.          |
| Send Cookies     | -b           | --cookie      | Sends cookies from a string or file. Format of string should be NAME=VALUE; another=anotherval. |
| Verbose Mode     | -v           | --verbose     | Provides more information (debug info).              |
| Silent Mode      | -s           | --silent      | Silent mode. Don't output anything.                  |
| Output to File   | -o           | --output      | Writes output to instead of stdout.                 |
| Pass a Custom Header | -H       | --header      | Passes a custom header to the server. To get JSON, use -H "Accept: application/json". |

Conclusions

We trust that this guide has provided you with a solid grasp of the fundamentals of cURL GET requests. However, as is the case with any skill, practice is crucial. We suggest dedicating some time to sending requests, and you'll gradually gain confidence with cURL.

If you're interested in utilizing proxies, check out our blog post on "cURL with Proxy". Additionally, we have a blog post covering "cURL with Python", offering insights into using the cURL command alongside Python code.

FAQ

What is the purpose of cURL GET?
In simple terms, a cURL GET request fetches data from a designated resource. It dispatches an HTTP request to a server and receives the response body, usually containing the content of a webpage or the requested data. Optionally, you can also retrieve the response headers, offering additional details such as content type, content length, server information, and more.

How do I use cURL for a GET request?
Performing a GET request with cURL is simple. You just need to open your terminal or command line and type 'curl' followed by the URL you want to request. Take a look at the following example:

bashCopy codecurl http://httpbin.org/get

This command sends a GET request to the provided URL and displays the HTML code of the webpage on the terminal.

How to get data in JSON format using cURL?
To begin, open your terminal or command line and enter the command 'curl.' To obtain data in JSON format, utilize the -H option followed by "Accept: application/json," along with the URL you wish to request. Here's an example:

bashCopy codecurl -H "Accept: application/json" http://httpbin.org/get