Error in regex formulation for web scraping in python

Question

I am trying to scrape some information from a website. I require 8 fields of information, I have got it for 5 fields, but 3 fields are always coming empty. I think there is some mistake with my regular expression formulation. I am doing it in python and I don't have to use BS. Here are the HTML fileds I need to scrape. This is example of one of the webpage.

enter code here

<td><span class="facultyName">John Matthew Falletta, MD</span>

<span class="primaryTitle">Professor of Pediatrics</span>

<span class="secondaryTitle">Professor in the School of Nursing</span>

<td><span class="label">Department:</span>
        &nbsp;&nbsp;
    </td><td>Pediatrics</td>

<td><span class="label">Division:</span>
        &nbsp;&nbsp;
    </td><td>Hematology/Oncology</td>

<td><span class="label">Address:</span></td><td>Box 2991<br>DUMC<br>Durham, NC &nbsp;27710   </td>

<td><span class="label">Phone:</span></td><td>
       (919)
       668-5111<br>

<td><span class="label">FAX:</span></td><td>                
        (919)
        688-5125</td>

Here is my code containing respective regular expressions for each type of tag:

enter code here

patFinderFullname = re.compile('<span class="facultyName">(.*)</span>')
patFinderPTitle = re.compile('<span class="primaryTitle">(.*)</span>')
patFinderSTitle = re.compile('<span class="secondaryTitle">(.*)</span>')
patFinderDepartment = re.compile('<span class="label">Department:</span>s+&nbsp;&nbsp;s+</td><td>(.*)</td>')
patFinderDivision = re.compile('<span class="label">Division:</span>s+&nbsp;&nbsp;s+</td><td>(.*)')

patFinderAddress = re.compile(' <span class="label">Address:</span>s+(.*)s+</td>')
patFinderPhone = re.compile('<span class="label">Phone:</span></td><td>s*(.*?)s*<br>')
patFinderFax = re.compile('<td><span class="label">FAX:</span>s+</td><td>s+(.*)</td>')

First five field results are coming correct, but the last three fields for Address, Phone and Fax are returning always empty. Can anyone point out what I am missing? Or what is wrong with the regular expressions for the last three fields. I have posted an earlier [1][question], but these problems arrived later to that, so I am asking it in a different question.

[1] : How to scrape html tags spread over multiple lines in python?

Answer 1

patFinderAddress = re.compile('<td><span class="label">Address:</span></td>.*?</td>'
patFinderPhone  = re.compile('<td><span class="label">Phone:</span>s*</td><td>s*^s*.*s*^s*.*<br>',re.M)
patFinderFax = re.compile('<td><span class="label">FAX:</span>s*</td><td>s*^s*.*s*^s*.*</td>',re.M)

Here's the some regexs that work with your data. The last two weren't working as the data spanned multiple lines. The first didn't work because it was wrong.

But, for html parsing, use an html parser as it's far more robust and gives you the data you want rather than this eyesore of html strings.