python - How to store data from web scraping poject

Question

#Background

I am currently playing with some web scraping project as I am learning python. I have a project which scrapes products with information about price etc using Selenium. Than I add every record to pandas DF, do some additional data manipulation and than store data in csv and upload to google drive. This runs every night

#Question itself

I would like to watch price changes, new products etc. Would you recommend, how to store data with date key, so there is option to flag new products etc? My idea is to store every load in one csv and add one column with "date_of_load"... But this seems noob_like... Maybe store data in PostrgreDB? I would like to start learning SQL, so I would try making my own DB.

Thanks for your ideas

Answer 1

As for me better to use NoSQL (Mongo) for this task. You can create JSON (data of prices) with keys are date. This can help you:

• https://www.mongodb.com/blog/post/getting-started-with-python-and-mongodb
• https://www.mongodb.com/python
• https://realpython.com/introduction-to-mongodb-and-python/
• https://www.google.com/search?&q=python+mongo