• Install Geckodriver and all other necessary packages

• Explore how Selenium webdriver and geckodriver works.

• Import all necessary libraries in the script.

• Open Instagram login page using driver. Use driver.get(url) to open instagram login page.

• Checkout unique identifiers for the input fields login and passwords.

• Explore how xPaths help locating various elements in a web page.

• Examples:

  • xpath for following 'Username input field' element would be //input[@name='username']

  

  • and xpath for the 'Save Post button' element will be //span[@class="wmtNn"]/div/div/button

  

• Discover login, password and submit elements and interact with them to achieve sign in. Use driver.find_element_by_xpath(xpath) to find the elements and use input_element.send_keys(“text”) to send text in the input boxes. Use submit_element.click() to achieve click action.

• Open various starter links for explore page/ tags page/ profile page using driver.get, note here while using driver.get function our page might reload and we can lose reference to previously stored elements in variables.

• Explore element scroll and click functionality to interact with buttons. This might be needed for two reasons, one selenium can't interact with an element if it's not in the view, and secondly, in modern web pages content loads dynamically, so scrolling down may trigger loading of additional content. We can also achieve scrolling by running javascript functions using python as shown below.

    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

• You may encounter some elements on which clicking directly is not supported. For those cases we can use the lambda function as shown below.

        follow_button = lambda: driver.find_element_by_xpath(xpath).click()
        follow_button()

• Explore send keys function to achieve comments functionality. In selenium we can use send keys functionality to emulate typing in a browser, which is quite useful to fill data in forms.
• During automation we can also store some data like post URLs, text, and profile details for further processing in CSVs or a NoSQL database like mongoDB. Now for this step we can structure the data gathering in a way that will help us in the next step while processing. It is preferred to use a mongoDB in this case because the relations between profiles and posts and between posts and comments is a one to many relationship. But we can also store this data in csv as well in such a way that it is easier for further processing. We can create 3 csvs, one with all the profile data like handle,follower count, following count, no of posts, and profile url/image. Another csv for posts which will contain for each post, no of likes, profile handle, media(image/video) url, the description of the post, date posted. We will also need one csv for storing comments as well. Now, the structure of storing data and storage type will depend on use cases and we can experiment with various permutations and combinations. This step will help you understand how databases are designed because we will be reverse engineering them in ways. So while you build scripts/scrapers for other websites as well, it will help you in understanding their database design as well.
• Check out some ML libraries or APIs to enhance engagement performance. This is a nice to have feature, and requires skills beyond scope of this project. But if you are interested here are some ideas to try. We can use image processing to detect content in an image and customize our comments and replies. We can also use text analysis using NLP to also detect intent of the comments/DMs and generate personalised comments/replies for them.

• Now we need to read URLs stored in CSV or the mongoDB database collected in previous milestones and explore various ways to request files from a server with those URLs. We can utilise urlretrieve function from urllib.request library to make requests on Instagram severs.

    urllib.request.urlretrieve(source_url, target_name)

• We can always store the data on our local machine, but often our storage may run out of space because image data is generally larger in size than text data. Instead of storing this data locally we can chose to store them on cloud, either in an AWS S3 bucket or gcp bucket. It is free at the starting because these services provide a good amount of storage in their free tier.
• It can so happen that urlretrieve may not fetch any result, which can happen due to several reasons like its expiration, or throttling, or due to unavailability etc. For these cases we may need to structure our code to handle it. In case of expiration/unavailability on the severs, we can store/mark those URLs with a flag. If the count of URLs with this flag as true continuously exceeds, we may need to trigger to stop the script because this is the case or throttling. To prevent this from happening, we need to add sleep statements between each request.
• Now we have different kinds of data like Profile Stats, Image data, Text data. We can now build/model various machine learning/deep learning/NLP projects post this project. This step is optional as far as this project goes.
• Explore how to handle exceptions in python. The dynamic nature of modern websites can be prone to some unexpected events which might trigger some exceptions which may lead to halting of script. So, to handle these exceptions we might need to use exception handlig to handle such cases.

try:
    //code which might throw an exception.
except:
    //what to do if an exception occurs. Hint - log it.
finally:
    //this code runs always.

• Explore how to log these errors for continuous development. Logging is also used to monitor health of the script which is generally running without supervision. So we need to add Log statements for various events in our script. That is why you might want to put log statements at various stages of automation.

• Create a Docker container for your application. It will make the deployment easier.
• Setup a cloud instance on GCP and activate it.
• Upload your files to the platform. You should simply use your GitHub repository here, since it’ll do the job by a simple git clone.
• Run your application on the platform.
• Here you might want to run the automation in headless mode.

options.headless=True