Gripper

A web-scraping toolbox for .NET.

View on GitHub

←Home About→ Features→ API→

Getting Started

The client code commands Gripper through the IWebClient interface that controls a single browser window. This guide shows how to install, register and configure for use in a .NET console application or hosted service.

Installation

Gripper is released as Gripper.WebClient on Nuget.

This package depends on:

  1. .NET 6
  2. The autogenerated Chrome Devtools Protocol (CDP) wrapper, Gripper.ChromeDevTools.

The Gripper.ChromeDevTools package has several other dependencies.

Note that the current version of Gripper.WebClient is a prerelease. When installing from the CLI, don’t forget the --prerelease flag. In the Visual Studio Nuget package manager, toggle the Prerelease checkbox.

From Nuget

The easiest way to get Gripper is directly from Nuget (info outputs omitted):

> dotnet new console -f net6.0 -n GripperDemo
The template "Console App" was created successfully.
Processing post-creation actions...
Running 'dotnet restore' on C:\Users\tomas\source\demos\GripperDemo\GripperDemo.csproj...
  Determining projects to restore...
  Restored C:\Users\tomas\source\demos\GripperDemo\GripperDemo.csproj (in 77 ms).
Restore succeeded.

> cd GripperDemo

> dotnet add package Gripper.WebClient --prerelease
  Determining projects to restore...
  Writing C:\Users\tomas\AppData\Local\Temp\tmp3442.tmp
info : PackageReference for package 'Gripper.WebClient' version '0.4.7-alpha' added to file 'C:\Users\tomas\source\demos\GripperDemo\GripperDemo.csproj'.

> dotnet run
Hello, World!

From Source

Gripper can be obtained directly from the repo and added as a solution dependency in one of the following ways:

Hosting

Gripper is designed as a service and hosting within an IHost environment should be the preferred way of deployment. Locally, the IWebClient can be resolved directly from an IServiceProvider.

The Gripper.WebClient.Extensions namespace provides extensions for adding Gripper into an IServiceCollection.

In any case, each time the client resolves the transient IWebClient service, a fresh browser window is launched and bound.

// BrowserAutomationAgent/program.cs

using Gripper.WebClient;
using Gripper.WebClient.Extensions;
using Microsoft.Extensions.DependencyInjection;

var serviceProvider = new ServiceCollection()
    .AddGripper() // Register all services + default settings.
    .AddLogging()
    .BuildServiceProvider();

var webClient = serviceProvider.GetRequiredService<IWebClient>(); // Fresh window driver.

var cts = new CancellationTokenSource(TimeSpan.FromSeconds(30));

await webClient.NavigateAsync(
    "https://tomaskrupka.github.io/Gripper/about",
    PollSettings.FrameDetectionDefault,
    cts.Token);

var linksCountResponse = await webClient.MainContext.ExecuteScriptAsync(
    "document.querySelectorAll('a').length",
    cts.Token);

Console.WriteLine($"Found {linksCountResponse.Result.Value} hyperlinks.");

Running the above code locally:

PS C:\Users\tomas\source\demos\BrowserAutomationAgent\BrowserAutomationAgent> dotnet run
Found 37 hyperlinks.

Configuration

Gripper is configured by passing a WebClientSettings instance, using the options pattern. The current range of available options is documented in the source.

This is best achieved using the AddGripper() extension. This extension will register the default implementation for all services necessary to run Gripper.

Default Configuration

The AddGripper() extension always configures Gripper with the default settings. When new WebClientSettings object is provided, its non-default members override these of the default settings, and the omitted members remain at their default values.

var serviceCollection = new ServiceCollection();
var settings = new WebClientSettings
{
    UseProxy = true,
    Proxy = new System.Net.WebProxy("127.0.0.1", 8080)
};

// Choose one 👇:
serviceCollection.AddGripper(); // Just default settings.
serviceCollection.AddGripper(settings);
serviceCollection.AddGripper(s => s.BrowserLocation = "path/to/chrome.exe");

serviceCollection.AddLogging(); // Don't forget logging.
var serviceProvider = serviceCollection.BuildServiceProvider();
var webClient = serviceProvider.GetRequiredService<IWebClient>(); // Browser window driver.

Logging

At runtime, Gripper will try to resolve the ILoggerFactory and ILogger<T> services. Therefore, logging services must be added to the ServiceProvider hosting Gripper.

To prevent collisions with other configurations accessing the same container, the AddGripper() extension does not register logging services.

Builders

The WebClientSettingsGenerator class provides out-of-the-box settings objects and convenience methods for creating new ones.

The WebClientSettingsExtensions provides extensions for editing existing configurations.