Use cURL to access password protected content via Script

I have a need to store some webpages to parse later and extract data for some enhanced contest stats and this script is the first step to log in, then pull those pages down.

Required tools: Linux OS, cURL , Internet Connection

First, you need to go to the webpage with the login form on it and determine where it is going to send you one you enter your username and password.  You can usually find this by viewing the Source of the page and then searching for Login.  The info you need here will be located in the "form" tag directly above it.  This tag will look something like this. In this example, you need the information in "action".

<form action="http://example.com/login.php" method="POST" name="login"></form>

Ok, we're ready to start.  We need to set up our cURL cookie jar to be able to store cookies for this domain, enter this command to do so.

curl --cookie-jar cjar --output /dev/null http://example.com/login.php

Now that the cookie jar is set up, we can login, you'll need to get the "name" of the login, password and any other variables passed to the url above.  You'll have to look in the source code to determine this, but it will be anything between the "form" tags above. In this example, we'll use login and password with no hidden variables that need to be passed.

curl --cookie cjar --cookie-jar cjar --data 'login=myuser@example.com' --data 'password=mypassword' --location  --output loginresult.html http://example.com/login.php

So in the above example here's what we are saying.
curl --cookie cjar --cookie-jar cjar - Use the cookie from the cookie jar we set up earlier
--data  - Here we have both the login and password that will be sent
--output - This will allow us to output the received page to your local server. You can view this to see if the result was successful  - You tell it the output file and the login URL.

Ok, now we're logged in, lets grab a page.

curl --cookie cjar  --output downloadedfile.php http://secure.example.com/downloadedfile.php

That's it, you should now have the "downloadedfile.php" downloaded to your server.