Docs / Handling Authentication in Web Scraping

Handling Authentication in Web Scraping

Overview

Many valuable data sources require authentication before allowing access to their content. This guide covers various techniques for handling authentication in your DataScrap Studio projects, from basic form logins to more complex authentication flows.

Understanding Authentication Types

Before implementing authentication, it’s important to understand the different methods websites use:

Form-Based Authentication

The most common type, where users enter credentials into a form:

  • Username/email and password fields
  • Submit button or form submission
  • Often redirects after successful login

After login, the website sets cookies to maintain your session:

  • Session cookies track your authenticated state
  • May expire after inactivity or a set time period
  • Can sometimes be transferred between scraping sessions

Token-Based Authentication

Modern websites often use JWT (JSON Web Tokens) or similar:

  • Tokens are typically stored in local storage or cookies
  • Must be included in request headers
  • May require periodic renewal

Multi-Factor Authentication (MFA)

Additional security layer beyond passwords:

  • Time-based codes (TOTP)
  • SMS verification
  • Email confirmation links
  • Hardware security keys

Basic Form Login

The simplest authentication method is automating the login form process:

Step 1: Configure the Login Process

  1. Create a new project in DataScrap Studio
  2. Navigate to the website’s login page
  3. Go to Project > Authentication > Form Login
  4. Click Record Login Sequence

Step 2: Record the Login Sequence

  1. Click on the username/email field
  2. Enter your credentials when prompted
  3. Click on the password field
  4. Enter your password when prompted
  5. Click the login/submit button
  6. Wait for the successful login (usually a redirect)
  7. Click Stop Recording

Step 3: Verify and Save

  1. Test the login sequence with Test Authentication
  2. If successful, you’ll see “Authentication Successful”
  3. Save the authentication profile with a descriptive name
  4. Enable Use Authentication in your project settings

Example: LinkedIn Login

Authentication Profile: LinkedIn
Steps:
1. Navigate to: https://www.linkedin.com/login
2. Fill field: #username with ${USERNAME}
3. Fill field: #password with ${PASSWORD}
4. Click: button[type="submit"]
5. Wait for: .feed-identity-module

For sites where form login is complex or changes frequently, cookie-based authentication is more reliable:

Method 1: Import Cookies from Browser

  1. Log in to the website manually in your regular browser
  2. Use a browser extension to export cookies (like “Cookie-Editor” for Chrome)
  3. In DataScrap Studio, go to Project > Authentication > Cookie Import
  4. Import the cookie file or paste the cookie JSON
  5. Test the authentication

Method 2: Capture Cookies After Login

  1. Configure form login as described above
  2. Enable Save Session Cookies in the authentication settings
  3. Run the login sequence once
  4. DataScrap Studio will store the cookies for future use
  5. Set Cookie Refresh Interval based on the website’s session duration
  • Store sensitive cookies securely using the built-in credential manager
  • Set appropriate refresh intervals to prevent session expiration
  • Use different cookie profiles for different accounts or websites
  • Monitor for cookie format changes after website updates

Token-Based Authentication

For modern web applications using token authentication:

Capturing Authentication Tokens

  1. Use your browser’s developer tools (F12) while logging in
  2. Watch the Network tab for the authentication request
  3. Note the response containing the token (usually in JSON format)
  4. Identify where the token is stored (localStorage, cookies, etc.)

Implementing in DataScrap Studio

  1. Go to Project > Authentication > Advanced > Custom Headers
  2. Add the appropriate authorization header:
    Authorization: Bearer ${TOKEN}
  3. Configure token refresh if needed:
    • Set Token Endpoint URL
    • Configure Refresh Parameters
    • Set Refresh Interval

Example: JWT Implementation

Authentication Type: JWT
Token Endpoint: https://api.example.com/auth/token
Request Type: POST
Request Body: {"refresh_token": "${REFRESH_TOKEN}"}
Token Path: $.access_token
Header Format: Authorization: Bearer ${TOKEN}
Refresh Before: 5 minutes

Handling Multi-Factor Authentication

MFA presents special challenges for automated scraping:

One-Time Setup Approach

For personal use with your own accounts:

  1. Complete MFA manually once
  2. Save the resulting long-lived session cookie
  3. Use this cookie for future scraping sessions
  4. Refresh when eventually expired

Time-Based One-Time Password (TOTP)

For accounts using authenticator apps:

  1. Store the TOTP secret in DataScrap Studio’s secure storage
  2. Go to Project > Authentication > Advanced > TOTP
  3. Enter the TOTP secret key
  4. Configure the login sequence to use the generated code:
    Fill field: #totp-code with ${TOTP_CODE}

Email Verification Handling

For sites that send verification emails:

  1. Configure email checking in Authentication > Email Verification
  2. Connect to your email account using IMAP settings
  3. Set up filters to identify verification emails
  4. Configure extraction of verification links or codes
  5. Complete the authentication flow using the extracted information

Secure Credential Management

Protect your authentication credentials within DataScrap Studio:

Using the Credential Manager

  1. Go to Tools > Credential Manager
  2. Add a new credential set with:
    • Descriptive name
    • Username/email
    • Password
    • Additional fields as needed
  3. Reference in projects using variables:
    ${CREDENTIALS.sitename.username}
${CREDENTIALS.sitename.password}

Environment Variables

For team environments or additional security:

  1. Store credentials as environment variables on your system
  2. Reference them in DataScrap Studio:
    ${ENV.SITE_USERNAME}
${ENV.SITE_PASSWORD}

Encryption Options

  1. Go to Settings > Security
  2. Enable Encrypt Stored Credentials
  3. Set a master password
  4. Configure automatic locking after inactivity

Troubleshooting Authentication Issues

Login Detection Problems

If DataScrap Studio can’t detect successful login:

  1. Go to Authentication > Success Detection
  2. Configure a specific element that only appears when logged in
  3. Set an appropriate timeout value
  4. Add alternative success indicators if needed

Frequent Session Expiration

If your session expires too quickly:

  1. Increase Cookie Refresh Interval
  2. Check for required activity to maintain sessions
  3. Add “heartbeat” requests to keep the session alive
  4. Look for session extenders (like “Remember Me” options)

Captcha Challenges

If login triggers CAPTCHA:

  1. Reduce login frequency
  2. Enable Humanized Behavior in browser settings
  3. Use the Captcha Solver extension if available
  4. Consider using pre-authenticated cookies instead

Case Study: E-commerce Account Scraping

Scenario

A user needs to extract their order history from an e-commerce platform that requires login and occasionally uses email verification.

Solution

  1. Authentication Setup:

    • Recorded form login sequence
    • Configured email verification handling
    • Stored cookies after successful authentication

  2. Extraction Configuration:

    • Navigated to order history page
    • Created selectors for order details
    • Set up pagination handling

  3. Maintenance Approach:

    • Weekly cookie refresh
    • Alert on authentication failures
    • Backup authentication method using API tokens

Best Practices

Security Considerations

  • Never share projects containing credentials
  • Use credential variables instead of hardcoded values
  • Enable encryption for stored authentication data
  • Regularly update passwords used in scraping projects

Performance Optimization

  • Minimize authentication requests
  • Cache authenticated sessions when possible
  • Use the same browser profile to maintain cookies
  • Implement efficient token refresh strategies
  • Only authenticate with accounts you own or have permission to access
  • Respect rate limits and terms of service
  • Consider using official APIs if available
  • Be aware of potential legal implications of automated access

Conclusion

Effective authentication handling is essential for accessing valuable data behind login screens. DataScrap Studio provides multiple approaches to handle various authentication methods, from simple form logins to complex multi-factor scenarios.

By understanding the authentication mechanisms used by your target websites and implementing the appropriate techniques, you can reliably extract data from authenticated sources while maintaining security and respecting website limitations.

Additional Resources

If you encounter complex authentication scenarios not covered in this guide, please contact our support team for personalized assistance.

Last updated: December 20, 2023

Was this page helpful?

Next First-Time Setup Guide