Web scraping functions in Power Query
Every day more and more information needs to be collected from the Internet for analysis, since it is the most extensive source of data. In Power Query there is not enough functionality to work with complex site structures. To expand the functionality in this area, I propose:
1) rework the site viewer - enable a mode similar to the debugging mode in Opera, Chrome or Mozilla, when you can select a site object in the visual part, and the path will be set in the code to access it.
2) expand the set of elements available for processing, in particular, obtaining statistical data from such a large resource as cbsd.gks.ru
3) add the following functions (please, complete with what you consider the most important):
- support for both http and https
- full support for http methods;
- processing of all types of headings
- replacement user-agent;
- Support for processing through proxy (HTTP, HTTPS and Socks including chaining) and VPN
- a convenient function to set the delay before another request, including randomly within the selected interval.