Page 57 - Hands-On Bug Hunting for Penetration Testers

Preparing for an Engagement Chapter 3

4DSBQZ BOE $VTUPN 1JQFMJOFT

TDSBQZ is a popular web-crawling framework for Python that allows you to create web
crawlers out of the box. It's a powerful general-purpose tool that, since it allows a lot of
customization, has naturally found its way into professional security workflows. Projects
such as XSScrapy, an XSS and SQLi scanning tool built on Scrapy, show the underlying
base code's adaptability. Unlike the Burp Suite Spider, whose virtue is that it integrates
easily with other Burp tools, and Striker, whose value comes in collecting DNS and
networking info from its default configuration, Scrapy's appeal is that it can be set up easily
and then customized to create any kind of data pipeline.

Manual Walkthroughs

If the app doesn't have a sitemap, and you don't want to use a scanner, you can still create a
layout of the site's structure by navigating through it, without having to take notes or
screenshots. Burp allows you to link your browser to the application's proxy, where it will
then keep a record of all the pages you visit as you step through the site. As you map the
site's attack surface, you can add or remove pages from the scope to ensure you control
what gets investigated with automated workflows.

Doing this manual-with-an-assist method can actually be preferable to using an automated
scanner. Besides being less noisy and less damaging to target servers, the manual method
lets you tightly control what gets considered in-scope and investigated.

First, connect your browser to the Burp proxy.

Portswigger provides support articles to help you. If you're using Chrome, you can follow
along with me here. Even though we're using Chrome, we're going to use the Burp support
article for Safari because the setting in question is in your Mac settings: IUUQT TVQQPSU
QPSUTXJHHFS OFU DVTUPNFS QPSUBM BSUJDMFT *OTUBMMJOH@
$POGJHVSJOH ZPVS #SPXTFS 4BGBSJ IUNM.
Once your browser is connected and on (and you've turned the Intercept function off), go
to IUUQ CVSQ .

If you do this through your Burp proxy, you'll be redirected to a page where you can
download the Burp certificate. We'll need the certificate to remove any security warnings
and allow our browser to install static assets:

[ 42 ]