Unable to extract table content from webpage using selenium

Question

I am trying to extract the tables generated by selecting "Branches", a city and a district from this site: https://www.acb.com.vn/wps/portal/en/atm So far, I have been able to write the code to parse through each city and district: from selenium.webdriver.support.ui import Select from selenium.webdriver import Chrome import pandas as pd import time webdriver = "chromedriver.exe" driver = Chrome(webdriver) driver.get('https://www.acb.com.vn/wps/portal/en/atm') branch_selector = driver.find_element_by_xpath('//*[@id="branch"]') branch_selector.click() city = Select(driver.find_element_by_id('cityId')) for i in range(len(city.options)): city.select_by_index(i) time.sleep(1) district = Select(driver.find_element_by_id('districtId')) for j in range(len(district.options)): district.select_by_index(j) time.sleep(1) try: find_btn = driver.find_element_by_xpath('//*[@id="frm-filter"]/div[3]/a[1]') find_btn.click() time.sleep(1) except: close_btn = driver.find_element_by_xpath('//*[@id="close-send-email"]/span[2]') close_btn.click() time.sleep(1) Now, I want to extract the table that's displayed in each iteration of the 2 loops. However, if you look at the HTML for the table, it does not make use of the "table" tag: So, how do I extract the table for each city-district pair? I tried the following: try: click_btn = driver.find_element_by_xpath('//*[@id="frm-filter"]/div[3]/a[1]') click_btn.click() time.sleep(1) table = driver.find_elements_by_class_name('tbody') for table_row in table: row = table_row.find_elements_by_class_name('row') print ([r.text for r in row]) except: close_btn = driver.find_element_by_xpath('//*[@id="close-send-email"]/span[2]') close_btn.click() time.sleep(1) But it prints a list of blank elements for each city-district pair, the length of the list being as many addresses are present in the table for the corresponding city-district pair: ['', '', '', ''] ['', '', '', ''] ['', '', '', ''] ['', '', '', ''] ['', '', '', ''] ['', '', '', ''] ['', '', '', '', ''] ['', '', '', '', ''] ['', '', '', '', ''] ['', '', '', '', ''] ['', '', '', '', ''] ['', '', '', '', ''] ['', '', '', ''] ['', ''] ['', ''] I also tried to access each element in each row of the table individually: try: find_btn = driver.find_element_by_xpath('//*[@id="frm-filter"]/div[3]/a[1]') find_btn.click() time.sleep(1) table = driver.find_elements_by_class_name('tbody') for table_row in table: row = table_row.find_elements_by_class_name('row') for element in row: time.sleep(1) Type.append(element.find_element_by_class_name('col type')) Address.append(element.find_element_by_class_name('col address')) District.append(element.find_element_by_class_name('col district')) Tel_Fax.append(element.find_element_by_class_name('col tel-fax')) Hours.append(element.find_element_by_class_name('col hours')) except: close_btn = driver.find_element_by_xpath('//*[@id="close-send-email"]/span[2]') close_btn.click() time.sleep(1) But this gives the following error: --------------------------------------------------------------------------- NoSuchElementException Traceback (most recent call last) in 39 ---> 40 Type.append(element.find_element_by_class_name('col type')) 41 Address.append(element.find_element_by_class_name('col address')) NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="col type"]"} Since it says css selector in the error, I tried the following: element.find_element_by_css_selector('div.col.type').text This outputs a blank string, ''. So, how do I do this? EDIT: The HTML of the table, for one district-city selection, is:

1

PGD Hai Bà Trưng

56-58-60 Hai Bà Trưng, P. Bến Nghé, Quan 1, Ho Chi Minh

1

(028) 6291 3690
(028) 6291 3691

07:00-16:30

Direction

2

PGD Đa Kao

45 Võ Thị Sáu, P. Đa Kao, Quan 1, Ho Chi Minh

1

(028) 6290 5980
(028) 6290 5981

07:30 – 16:30

Direction

3

PGD Nguyễn Công Trứ

74 - 76 Nguyễn Công Trứ, P. Nguyễn Thái Bình, Quan 1, Ho Chi Minh

1

(028) 3914 4470
(028) 3914 4471

07:30 – 16:30

Direction

4

PGD Lê Lợi

72 Lê Lợi, P. Bến Thành, Quận 1, TP.Hồ Chí Minh

1

(028) 3821 4619
(028) 3821 4618

07:00-16:30

Direction

5

CN Sài Gòn

41 Mạc Đỉnh Chi, P. Đakao, Quan 1, Ho Chi Minh

1

(028) 3824 3770
(028) 3824 3946

07:30 – 16:30

Direction

6

PGD Nguyễn Thái Bình

176 – 178 Ký Con, P. Nguyễn Thái Bình, Quan 1, Ho Chi Minh

1

(028) 3915 1310
(028) 3915 1311

07:30 – 16:30

Direction

7

PGD Bến Chương Dương

328 Võ Văn Kiệt, phường Cô Giang, Quận 1, Tp.HCM

1

(028) 3837 0586
(028) 3837 0584

7h30-16h30

Direction

8

PGD Trần Khắc Chân

48-50 Nguyễn Hữu Cầu, P.Tân Định, Q.1, TP.HCM

1

(028) 3820 9990
(028) 3526 7738

07:30 -16:30

Direction

9

PGD Cống Quỳnh

106 108 Cống Quỳnh, P. Nguyễn Cư Trinh, Q.1

1

(028) 38385464
(028) 3925 6645

07:30 -16:30

Direction

10

CN Bến Thành

96 Lý Tự Trọng, P. Bến Thành, Quan 1, Ho Chi Minh

1

(028) 3825 7949
(028) 3825 7950

07:30-16:30

Direction

11

PGD Tân Định

261 Trần Quang Khải, Phường Tân Định, Quận 1, TP.HCM

1

(028) 3848 0520

07:30 - 16:30

Direction

12

PGD Nguyễn Du

Tầng hầm 1, tầng trệt, tầng lửng và tầng 2 tòa nhà 480 đường Nguyễn Thị Minh Khai, Phường 2, Quận 3, TP.Hồ Chí Minh

1

(028) 35218626
(028) 35218627

07:30 -16:30

Direction

Millie Anne Volante · Answer

Your problem might be sames as https://stackoverflow.com/questions/20888592/gettext-method-of-selenium-chrome-driver-sometimes-returns-an-empty-string
But upon initial checking on the attributes of your element, I would suggest to use "outerText" attribute. You can check your element's properties in developer's tool Console by typing $x("your_xpath").

Unable to extract table content from webpage using selenium

One Answer

Add your own answers!

Ask a Question