Python : find_all() return an empty list

Stack Overflow Asked by Bastien on December 17, 2020

I’m trying to make a bot that send me an email once a new product is online on a website.

I tried to do that with requests and beautifulSoup.

This is my code :

import requests
from bs4 import BeautifulSoup

URL = '[]=207&price_from=0&price_to=15&order=newest_first'

headers = {'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36"}

page = requests.get(URL, headers=headers)

soup = BeautifulSoup(page.content, 'html.parser')

products = soup.find_all("div", class_="c-box")


Next, I’ll want to compare the number of products before and after my new request in a loop.

But when I try to see the number of products that I found, I get an empty list : []

I don’t know how to fix that …

The div that I use is in others div, I don’t know if it has a relation

Thanks by advance

2 Answers

You should always check the data.

Convert your BeautifulSoup object to string with soup.decode('utf-8') and write it on a file. Then check what you get from the website. In this case, there is no element with c-box class.

You should use selenium instead of requests.

Answered by eren on December 17, 2020

You have problem with the website that you are trying to parse.

The website in your code generates elements you are looking for(div.c-box) after the website is fully loaded, using javascript, at the client-side. So it's like:

Browser gets HTML source from server --(1)--> JS files loaded as browser loads html source --> JS files add elements to the HTML source --(2)--> Those elements are loaded to the browser

You cannot fetch the data you want by requests.get because requests.get method can only get HTML source at point (1), but the website loads the data at (2) point. To fetch such data, you should use automated browser modules such as selenium.

Answered by cylee on December 17, 2020

Add your own answers!

Related Questions

how to use the “SPI_SETMOUSETRAILS” Parameters

1  Asked on November 27, 2021 by mingy


Detect pairs after simple frequency

2  Asked on November 27, 2021


Mailchimp API limit in post request

1  Asked on November 27, 2021


Python subprocess.check_output() for general shell command

1  Asked on November 27, 2021 by user13977273


Ajax returns GET request instead of POST

3  Asked on November 24, 2021 by zexar221


SIGABRT on std::ifstream close

1  Asked on November 24, 2021 by c0d3-m4513r


Python while loop counter ignores limit

2  Asked on November 24, 2021 by sh12030


Can’t access Pandas

2  Asked on November 24, 2021 by sash


HTML CSS style: hiding an HTML container and its HTML child controls

1  Asked on November 24, 2021 by deveric69


Ask a Question

Get help from others!

© 2022 All rights reserved. Sites we Love: PCI Database, MenuIva, UKBizDB, Menu Kuliner, Sharing RPP