1) screaming frog
screaming frog is a well known tool in the SEO industry, it has a pretty UI and will spider a site and find all the urls (it also provides other information that may be of use, eg redirect codes, url parameters, etc)

2) website crawler software kali Linux ( skipfish
skipfish -YO -o ~/Desktop/folder http://192.168.x.x
skipfish is included in kali linux and will spider a site for you (and can also test for various vulnerable parameters and configurations) – using the -O flag will tell skipifsh not to submit any forms, and -Y will tell skipfishn not to fuzz directories
This is very informative and well detailed. That is great and very helpful for hacking beginners. Thanks!