Yes, remote work is becoming increasingly popular. Remote work has been shown to increase productivity, save businesses money, and provide workers with more flexibility and control over their work-life balance. Let's build a job board web scraper to automate our job search to help us find work from jobs.

The use of job board scraping is becoming increasingly popular due to the ever-growing number of job postings available online. We will be creating a script in Python to search Google Job board. See the complete script on Github.

Tools Required

You'll need Python 2.7+ and some packages to get started. Once you install Python, you can run the following command to install the necessary packages.

pip install requests bs4 lxml

You have to use a specify query when searching on Google jobs board. You can specify the location and filter for remote jobs by using the keywords+ work from home. For example, if you want to search for a remote nurse position you have to use the terms nurse+work+from+home

An example of the complete URL will be https://www.google.com/search?q=nurse+work+from+home&ibp=htl;jobs

So our script will have to build this URL and grab the page source.

import requests
from bs4 import BeautifulSoup

keywords = "nurse+work+from+home"

# build the url
URL = f"https://www.google.com/search?q={keywords}&ibp=htl;jobs"

# make the request to get html
resp = requests.get(URL)
html = resp.content

Next, we need to parse the page for the jobs. That is where beautifulsoup comes in. It is an excellent library for parsing html code. You can right click the page content to see what to parse.

Then you can select the job title and get the selector to that job title.

Lets pass in the html content into beautifulsoup

# create a results variable to save the values
results = {"jobs": []}

# load the html into beautiful soup
soup = BeautifulSoup(html, "lxml")

Grab the job list using selector in beautifulsoup

# grab the job list
job_list_div = soup.find('div', {"aria-label": "Jobs list"})
# grab the un ordered list
job_ul = job_list_div.find('ul')
# get all the list items
job_list_items = job_ul.find_all('li', recursive=False)

Then we iterate through the list of divs to get the job title, location and perks. While printing out the job content.

# parse each list item
for job_list_item in job_list_items:
    job = {
        "perks": [],
        "position": job_list_item.find("h2").text
    }

    # grab the job title
    job["position"] = job_list_item.find("h2").text

    # get div with company name
    company_div = job_list_item.find("div", {"class": "vNEEBe"})
    job["company"] = company_div.text

    # get div with location
    location_div = job_list_item.find("div", {"class": "Qk80Jf"})
    job["location"] = location_div.text

    # get perks
    perks_divs = job_list_item.find_all("div", {"class": "I2Cbhb"})
    for perks_div in perks_divs:
        job["perks"].append(perks_div.text)

    print(job)

Then finally the results that the script produces:

[{'perks': ['3 days ago', 'Work from home', 'Full-time', 'No degree mentioned', '3 days ago', 'Work from home', 'Full-time', 'No degree mentioned'], 'position': 'TELE-TRIAGE REGISTERED NURSE (Remote)', 'company': 'BrightSpring Health Services', 'location': 'Anywhere'},
{'perks': ['2 days ago', 'Work from home', 'Full-time', '2 days ago', 'Work from home', 'Full-time'], 'position': 'Registered Nurse Oncology remote telephone', 'company': 'Inova Health System', 'location': 'Anywhere'},
{'perks': ['7 days ago', 'Work from home', 'Full-time', 'No degree mentioned', '7 days ago', 'Work from home', 'Full-time', 'No degree mentioned'], 'position': 'Virtual Remote-Patient Monitoring Registered Nurse (RN)', 'company': 'Orah', 'location': 'Anywhere'}]

There you have it. This script is pretty simple and grabs only the job title, company, and perks. This script can be improved and updated to grab the job descriptions.

Maybe the script can even automatically apply to the position and submit your resume. 😎 If you're looking for a scalable API service to automate the scraping requests checkout jobs boards API. Its even easier, feature rich, and extracts more information. Check out how simple it is to call the API is.

import requests
url = "https://api.serply.io/v1/job/search/q=nurse+practitioner"
headers = {"X-Api-Key": "YOUR_API_KEY"}
response = requests.request("GET", url, headers=headers)
print(response.json())

Here are the examples that it returns.

{
	"jobs": [
		{
			"highlights": [
				"Master’s Degree required OR commensurate experience and satisfactory completion of NP licensure",
				"Current RN and NP licensure in state of practice to include prescription authority or the ability to obtain prescriptive authority",
				"Board certified by the AANP, ANCC or the AACCN in a Matrix approved specialty",
				"Have a current BLS, ACLS or CPR certification"
			],
			"description": {
				"summary": "Matrix Medical Network is seeking a Nurse Practitioner Family Practice for a per diem / prn job in Sterling, Virginia.\n\nJob Description & Requirements",
				"detail": "Matrix Medical Network is seeking a Nurse Practitioner Family Practice for a per diem / prn job in Sterling, Virginia.\n\nJob Description & Requirements • Specialty: Family Practice\n• Discipline: Nurse Practitioner\n• Start Date: ASAP\n• Duration: Ongoing\n• Shift: flexible\n• Employment Type: Per Diem / PRN\n\nOverview\n\nNurse Practitioner – PRN\n\nHome Risk Assessments\n\nAbout Us\n\nMatrix Medical Network offers a broad range of clinical services and proven expertise that give primary care providers and the at-risk health plan members we visit with every day the tools and knowledge to better manage their health at home. With deep roots in clinical assessment and care management services, our national network of clinicians break through traditional barriers to care by meeting those members where they are. We help older adults and other at-risk individuals enjoy a better quality of care, experience improved health outcomes, and identify chronic conditions that may otherwise go undiagnosed.\n\nOur job opportunities allow you to leverage your expertise and compassion and make a direct impact to the health and well-being of others. Join our team and be rewarded by competitive compensation while making a difference in your community!\n\nMatrix Medical Network is proud to be a Diversity, Equity, Inclusion and Accountability Employer\n\nResponsibilities\n\nAbout the Position:\n\nDuring a visit that can last up to one hour, Matrix providers review and observe a member’s current health, medical history, medication adherence, social environment and other risks. This provides unmatched insight into a member’s overall health and well-being that can be difficult to capture during routine office visits. The Matrix Comprehensive Health Assessment helps to improve quality of care and allows us to potentially close multiple care gaps with a single visit.\n\nNurse Practitioner - PRN - Home Risk Assessments\n\nType: PRN-Flexible-Nurse Practitioner\n\nLocation: Home, SNF, Telehealth and other community environments in Sterling, VA\n\nHours: Days, Evening, Nights, weekend, Flexible Scheduling\n\nPAY: $95.00 or $110 depending on county per completed Assessment, 25 hours or more/month-Flexible Scheduling\n\nMileage Reimbursement paid for all miles when 20 or more completed assessments/month, No show payment for 10 or more booked assessments/month\n\nMatrix Provider – What To Expect\n• Make a difference in people’s lives by conducting Adult/Geriatric Assessments, medical history, diagnosis and treatment, health education, physician referrals, case management referrals, follow-up and clear documentation according to Matrix guidelines and protocols\n• Work collaboratively with physicians, case managers, social workers, family members, key caregivers and any appropriate ancillary medical personnel as appropriate\n• Collaborate with Primary Care Physician (PCP) on patient education, provide follow-up\n• A nationwide network of advanced practice providers to build and maintain relationships\n•
			},
			"link": "https://www.linkedin.com/jobs/view/per-diem-prn-nurse-practitioner-family-practice-covid19-%2495-110-per-per-visit-at-vivian-health-3459331245?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic",
			"is_remote": false,
			"is_hybrid": false,
			"logo": {
				"is_image": true,
				"src": "data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=="
			},
			"position": "Per Diem / PRN Nurse Practitioner - Family Practice - COVID19 - $95-110 per...",
			"employer": "Vivian Health",
			"location": "Sterling, VA",
			"source": "LinkedIn",
			"perks": [
				"22 hours ago",
				"Part-time",
				"Health insurance"
			]
		},
		...
	],
	"metadata": {
		"data_is_nearby_query": false,
		"data_query_location": "",
		"data_num_results_in_fetch": 10
	},
	"ts": 2.1689088344573975,
	"device_region": "US",
	"device_type": null
}

Scraping Google Jobs Board

Tools Required

Related Posts