
~~Shoreline was founded by Anurag Gupta in February 2019. Anurag was the VP and GM for Analytics and Database Services at Amazon Web Services (AWS). In addition to Anurag, the Shoreline team has strong operational experience from AWS, Apple and Cisco, where they managed large scale cloud applications and infrastructure. Shoreline is headquartered in Redwood City, California.~~

~~## Customer Problems Addressed by Shoreline~~

~~A significant majority of service related incidents in the cloud are well understood and can be addressed with automated remediation. Unfortunately, operators at many businesses with applications and services in public clouds such as AWS, GCP or Azure, spend a majority of their time addressing incidents manually, causing unnecessary ticket-related fatigue in operations teams. In addition, the mean-time-to-repair for many of the incidents remains high, breaking any customer related SLAs and internal SLOs.~~

~~Shoreline’s automated remediation platform increases service availability and reduces load on the operations teams. Shoreline’s platform takes care of most common issues quickly and automatically. In addition, operators can use Shoreline to debug and find the root cause for those occasional “first-time” issues. Shoreline does this by using actions that are easy for operators to create, and that are executed automatically by triggers based on processing of high resolution metrics collected in real time from customers’ applications and infrastructure.~~

## Typical Shoreline Customer

A typical Shoreline customer is a company that is running a large share or all of its digital services and applications in one or more more public clouds like Amazon’s AWS, Microsoft’s Azure or Google’s GCP. The company may have compute instances (nodes) ranging from a few hundred to a few thousand in these public clouds, supporting thousands of services end users. Such a company would use Shoreline to remediate issues automatically and quickly by deploying Shoreline’s agents on their nodes and using the Shoreline suite of SaaS applications to manage their services.

## How the Shoreline Solution Works

**Efficient agents with built-in automation** - agents are installed by the customers on their compute nodes. These agents scrape metrics every second from infrastructure and application, compresses them and stores them locally. To ensure the metrics are stored in reliable storage, the agents also “push“ these compressed metrics out to Shoreline service running in the cloud. The agents can also execute actions (snippets of code) that can take corrective actions on the node.

**Monitoring for remediation** - monitoring is done through a web-based graphical user interface. It helps users visually understand their infrastructure/services and the underlying metrics collected from them. Users can also see and monitor alarms in a single place. Like metrics, any alarms that have already been fired can be filtered by resources, services and by different time window.

**Remediation Actions** - users can create specific actions using the Shoreline Op Language, a language purpose built to perform specific actions fleet-wide or on specific resources based on metrics and associated alarms they generate.

**Bots** - bots allow users to connect named alarms to named actions to create automated remediations - "bots." This feature enables Shoreline to detect and remediate issues quickly, without human intervention

## Architecture

**Hardware** - all system, security, and network infrastructure are hosted in Amazon Web Services (AWS).

**Software** - the Shoreline agent (collects metrics and performs actions) and Saas service (manages metrics and performs operational activities) are proprietary and open source.

## Security

**Physical** - the Shoreline system and supporting infrastructure is hosted on AWS. As such, AWS is responsible for the physical security controls for the system.

**Network** - firewall systems are in place to filter unauthorized inbound network traffic from the Internet and deny any type of network connection that is not explicitly authorized. Network address translation (NAT) functionality is utilized to manage internal IP addresses. Administrative access to the firewall is restricted to authorized employees.

**Access** - Shoreline uses role-based security architecture and requires users of the system to be identified and authenticated prior to the use of any system resources. Resources are protected through the use of native system security and add-on software products and services that identify and authenticate users and validate access requests against the users’ authorized roles in access control lists. Employees and approved vendor personnel sign on to Shoreline infrastructure and services using either SSO or VPN, combined with two factor authentication.

## Operations

**Backups** - all customer data is backed up every hour to Amazon’s S3 storage service. All data is encrypted at rest and in transit. If any backups fail, personnel are notified appropriately.

**Availability** - Shoreline monitors the capacity utilization of physical and computing infrastructure both internally and for customers to ensure that service delivery matches customer expectations. Shoreline evaluates the need for additional infrastructure capacity in response to growth of existing customers and/or the addition of new customers.

## Technical Overview (TODO)

### Agents

A Shoreline Agent is a small piece of software that should be installed on every of software that is designed to run as a singleton instance on each node (VM/bare metal) that a customer wants to be managed by Shoreline. Customer should be able to use Shoreline Agent on a multitude of platforms across cloud providers in the most optimal packaging for their environment.


About Shoreline


## Tooling

### Supported operating systems

- x86, 64 bit
- Linux: Centos, Ubuntu
- macOS
- Windows: not supported at this time

### Accessing Op

- Option 1 - cURL the latest release, make the executable, and copy to your path (or add to another directory and add that to your path)
- Option 2 - download from GitHub, make the executable, and copy to your path (or add to another directory and add that to your path)
- Option 3 - download from the Shoreline website, make the executable, and copy to your path (or add to another directory and add that to your path)

### Configuration

User specific configuration is stored in .ops.yaml in the user home directory.  The configuration file allows users to specify the following settings:

- Debug mode
- Output raw JSON
- CLI output table default format
- Additional formatting options (i.e. strings, numbers, time, labels)
- Aliases
- Linux commands

More details can be found here (external link needs to be added).

### Access

Access to the Shoreline Op server is initiated via the AUTH command in the CLI:

`op> auth <server url>`

The auth command takes you to your organization’s identity management service where you enter your credentials to login into the Shoreline Op server.

![](/assets/images/articles/v1/op-language/access-diagram.png)

### How to Execute Commands

To start using Op, simply run the Op command.  This starts the Op interactive CLI shell which is the primary interface for operators.

`op>`

You can also execute Op commands from text files and stdin.  They are standard shell commands and can be natively invoked from existing Bash scripts and shells.

You can set it to verbose mode for detailed explanations by using the -v flag.  We recommend that you use verbose mode when starting out with Op to accelerate your ramp-up and also to assist with debugging.  Verbose output is written out to stdout.


Getting Started

Metrics


A Resource refers to a specific host, pod, or container throughout the Shoreline platform. It may also refer to a more distinct entity, such as a virtual machine or a database instance.

Resource types are platform and provider agnostic. So whether it's a fleet of pods and containers in Kubernetes, stand-alone hosts, or many virtual machines, Shoreline can control those Resources within AWS, GCP, or Azure.

## Discovering Resources

Operators need to understand how a workload is mapped to underlying Resources.

<note>

Shoreline monitors all your environment's Resources via [Agents](/t/agent) installed on each host node. You can view your monitored Resources through the [UI](/t/ui) or by downloading the [CLI](/t/cli) through the [Configuration](/t/configuration) page.
</note>

<modes>
<mode type="cli">

The [CLI](/t/cli) contains a handful of [Op](/t/op) commands to analyze and monitor your Resources.

- `host`: List all hosts.

<op>host</op>

```
 ID | TYPE | NAME                | REGION    | AZ
 1  | HOST | i-064246cabcbb73067 | us-east-1 | us-east-1b
 2  | HOST | i-0812b3fd801632ecc | us-east-1 | us-east-1b
 3  | HOST | i-0189e804e46578b34 | us-east-1 | us-east-1a
```

- `container`: List all containers.

<op>container</op>

```
 ID | TYPE      | NAME                                               | REGION    | AZ
 13 | CONTAINER | test10-cust.shoreline-2cscc.shoreline              | us-east-1 | us-east-1a
 14 | CONTAINER | monitoring.test10c-prometheus-0.prometheus-server  | us-east-1 | us-east-1a
 15 | CONTAINER | monitoring.test10c-node-exporter-w5r42.prometheus  | us-east-1 | us-east-1a
 16 | CONTAINER | kube-system.kube-proxy-6trzf.kube-proxy            | us-east-1 | us-east-1a
 17 | CONTAINER | kube-system.coredns-7d74b564bd-lv8nh.coredns       | us-east-1 | us-east-1a
```

- `pod`: List all pods.

<op>pod</op>

```
 ID | TYPE | NAME                                                  | REGION    | AZ
 4  | POD  | test10-cust.shoreline-2cscc                           | us-east-1 | us-east-1a
 5  | POD  | monitoring.test10c-prometheus-0                       | us-east-1 | us-east-1a
 6  | POD  | monitoring.test10c-node-exporter-w5r42                | us-east-1 | us-east-1a
 7  | POD  | kube-system.kube-proxy-6trzf                          | us-east-1 | us-east-1a
 8  | POD  | kube-system.coredns-7d74b564bd-lv8nh                  | us-east-1 | us-east-1a
```

To list all your Resources combine multiple Resource types with the [union](/op/functions/union) function.

<op>host union pod union container</op>

```
 ID | TYPE      | NAME                                 | REGION    | AZ
 1  | HOST      | i-064246cabcbb73067                  | us-east-1 | us-east-1b
 2  | HOST      | i-0812b3fd801632ecc                  | us-east-1 | us-east-1b
 4  | POD       | test10-cust.shoreline-2cscc          | us-east-1 | us-east-1a
 5  | POD       | monitoring.test10c-prometheus-0      | us-east-1 | us-east-1a
 13 | CONTAINER | test10-cust.shoreline-2cscc          | us-east-1 | us-east-1a
```

</mode>
<mode type="ui">

The [Shoreline UI](/t/ui) dashboard displays all monitored Resources under the `Resources Monitored` section.

![ui-home-resources](/assets/images/articles/resources/ui-home-resources.png)

By default, this collection groups your Resources by `type`. The first card always shows _all_ Resources. The rest of the panel adds Resource groups defined by the **Group by** option. For example, above we see the collection of `containers`, `pods`, and `hosts`.

The **Group by** options in this environment include `Resource Type`, `Region`, `Az`, `App`, and `Namespace`.

![ui-home-resources-group-by](/assets/images/articles/resources/ui-home-resources-group-by.png)

<note float="right" type="tip">

You can also reach the [Resources Page](/ui/resources) through the primary navigation menu at the top of the page.
</note>

Clicking on a Resource card opens a new browser tab to the [Resources Page](/ui/resources) and automatically filters the **Resources List** to match _only_ the selected Resource grouping.

For example, clicking on the **Us-East-1a** Resources card from the home page shows a list of all our Resources in the `us-east-1a` availability zone.

![ui-resources-page](/assets/images/articles/resources/ui-resources-page.png)
</mode>
</modes>

## Resource Relationships

Resources are also aware of their relationships with other Resources.

<modes>
<mode type="cli">

By piping multiple Resource queries into one statement you can traverse your Resource hierarchies.

<note type="tip">

The [Op pipe operator](/t/op/pipe) (`|`) passes the result of the previous command to the following command, similar to a pipe operator in \*nix shell environments.

</note>

Use the `host` command and [pipe](/t/op/pipe) the result to the `.pod` command to retrieve the Kubernetes pods that are scheduled to each host:

<op>host | .pod</op>

```
 ID | TYPE | NAME                                                  | REGION    | AZ
 24 | POD  | test10-cust.shoreline-tm5lh                           | us-east-1 | us-east-1b
 26 | POD  | monitoring.test10c-prometheus-1                       | us-east-1 | us-east-1b
 28 | POD  | monitoring.test10c-node-exporter-j6d8r                | us-east-1 | us-east-1b
 30 | POD  | kube-system.kube-proxy-99c5p                          | us-east-1 | us-east-1b
 32 | POD  | kube-system.coredns-7d74b564bd-b849k                  | us-east-1 | us-east-1b
```

Taking that a step further, pipe the above result to the `.container` command to retrieve all containers in each pod scheduled to each host:

<op>host | .pod | .container</op>

```
 ID | TYPE      | NAME                                                   | REGION    | AZ
 44 | CONTAINER | test10-cust.shoreline-tm5lh.shoreline                  | us-east-1 | us-east-1b
 46 | CONTAINER | monitoring.test10c-prometheus-1.prometheus-server      | us-east-1 | us-east-1b
 47 | CONTAINER | monitoring.test10c-node-exporter-j6d8r.prometheus-node | us-east-1 | us-east-1b
 48 | CONTAINER | kube-system.kube-proxy-99c5p.kube-proxy                | us-east-1 | us-east-1b
 49 | CONTAINER | kube-system.coredns-7d74b564bd-b849k.coredns           | us-east-1 | us-east-1b
```

To avoid repetition and retain valid command combinations you can use the `resource` command to create a named resource query alias.

For example, we're creating a named resource query called `app_containers` here.

<op>resource <span className={'highlight'}>app_containers</span> = host | .pod | .container</op>

```c
Created resource 'app_containers'.
```

The `app_containers` command is now shorthand for `host | .pod | .container` and returns the same result:

<op>app_containers</op>

```
 ID | TYPE      | NAME                                                   | REGION    | AZ
 44 | CONTAINER | test10-cust.shoreline-tm5lh.shoreline                  | us-east-1 | us-east-1b
 46 | CONTAINER | monitoring.test10c-prometheus-1.prometheus-server      | us-east-1 | us-east-1b
 47 | CONTAINER | monitoring.test10c-node-exporter-j6d8r.prometheus-node | us-east-1 | us-east-1b
 48 | CONTAINER | kube-system.kube-proxy-99c5p.kube-proxy                | us-east-1 | us-east-1b
 49 | CONTAINER | kube-system.coredns-7d74b564bd-b849k.coredns           | us-east-1 | us-east-1b
```

The `resource` command illustrates the real power of [Op](/t/op).  With [Op](/t/op), you can build up robust commands layer by layer.

<note type="warning">

Users cannot modify system reserved resource names or symbols such as `host`, `pod`, `container`, and `resources`.

</note>

</mode>
<mode type="ui">

The [Resources UI](/t/ui/resources) displays all discovered Resources in your environment. To filter resources, select an appropriate card within the **Resources Monitored** panel.

The Resource list at the bottom is filtered to show all pods by selecting the **POD** card filters.

![ui-resources-pod-card](/assets/images/articles/resources/ui-resources-pod-card.png)

</mode>
</modes>

<modes>
<mode type="cli">

## Filtering Resources

Let's expand on the `app_containers` query created above in [Resource Relationships](#resource-relationships). You can filter the returned resources in [Op](/t/op) by either **type** or **tag.**

For example, to get all of our containers in `us-east-1a` Availability Zone (AZ) we would run the following:

<op>app_containers | <span className={'highlight'}>az = "us-east-1a"</span></op>

```
 ID | TYPE      | NAME                                                   | REGION    | AZ
 13 | CONTAINER | test10-cust.shoreline-2cscc.shoreline                  | us-east-1 | us-east-1a
 14 | CONTAINER | monitoring.test10c-prometheus-0.prometheus-server      | us-east-1 | us-east-1a
 15 | CONTAINER | monitoring.test10c-node-exporter-w5r42.prometheus-node | us-east-1 | us-east-1a
 16 | CONTAINER | kube-system.kube-proxy-6trzf.kube-proxy                | us-east-1 | us-east-1a
 17 | CONTAINER | kube-system.coredns-7d74b564bd-lv8nh.coredns           | us-east-1 | us-east-1a
```

Inverting the equality comparator allows you to find all containers outside of the `us-east-1a` AZ:

<op>app_containers | az <span className={'highlight'}>!=</span> "us-east-1a"</op>

```
 ID | TYPE      | NAME                                                   | REGION    | AZ
 44 | CONTAINER | test10-cust.shoreline-tm5lh.shoreline                  | us-east-1 | us-east-1b
 46 | CONTAINER | monitoring.test10c-prometheus-1.prometheus-server      | us-east-1 | us-east-1b
 47 | CONTAINER | monitoring.test10c-node-exporter-j6d8r.prometheus-node | us-east-1 | us-east-1b
 48 | CONTAINER | kube-system.kube-proxy-99c5p.kube-proxy                | us-east-1 | us-east-1b
 49 | CONTAINER | kube-system.coredns-7d74b564bd-b849k.coredns           | us-east-1 | us-east-1b
```

Combining multiple **tag** strings into an array acts as an "OR" operation. Below we're seeking all containers in either the `us-east-1a` or `us-east-1b` AZs:

<op>app_containers | az = <span className={'highlight'}>["us-east-1a", "us-east-1b"]</span></op>

```
 ID | TYPE      | NAME                                                   | REGION    | AZ
 44 | CONTAINER | test10-cust.shoreline-tm5lh.shoreline                  | us-east-1 | us-east-1b
 46 | CONTAINER | monitoring.test10c-prometheus-1.prometheus-server      | us-east-1 | us-east-1b
 13 | CONTAINER | test10-cust.shoreline-2cscc.shoreline                  | us-east-1 | us-east-1a
 14 | CONTAINER | monitoring.test10c-prometheus-0.prometheus-server      | us-east-1 | us-east-1a
 15 | CONTAINER | monitoring.test10c-node-exporter-w5r42.prometheus-node | us-east-1 | us-east-1a
```

As discussed in [Resource Relationships](#resource-relationships) your resources are aware of their relationships. Using that internal awareness with piped commands provides some powerful capabilities.

For example, you can use the following series of commands to drill down through the data set one step at a time. Beginning at the highest of `host`:

- Find all pods within the host.
- Find all containers within the pods.
- Reduce the pods to only those in AZ `us-east-1a`.
- Finally, of those filtered pods, find those with a `container_name` of `shoreline`.

<op>host | .pod | .container | az = "us-east-1a" | <span className={'highlight'}>container_name = "shoreline"</span></op>

```
 ID | TYPE      | NAME                                  | REGION    | AZ
 13 | CONTAINER | test10-cust.shoreline-2cscc.shoreline | us-east-1 | us-east-1a
```

</mode>
</modes>

The [Op Language](/t/op) is capable of many more powerful capabilities beyond this brief overview of Resources. Check out the [Op Overview](/t/op), [Op Commands](/t/op/commands), and pre-built [Op Packs](/t/op/packs).


Resources


Alarms specify one or more [Metric](/t/metric) thresholds or custom [Resource](/t/resource) queries to check frequently. The Alarm is raised based on custom thresholds or shell commands you define, which informs any connected [Bot](/t/bot) to trigger remedial [Actions](/t/action).

<note>

Shoreline includes dozens of built-in [Metrics](/t/metric) on which to base your Alarms. You can also combine multiple [Metrics](/t/metric) into [Metric Sets](/t/metric-set) for monitoring many [Metrics](/t/metric) at once. You can even create your own [Derived Metrics](/t/derived-metric) if none of the built-in options meet your needs.

</note>

## Create an Alarm

Alarms can be created using either the [CLI](/t/cli) or the [Configuration UI](/t/ui/configuration).

<modes>

<mode type="cli">

To create a basic shell script Alarm with [Op](/t/cli) use the following syntax: `alarm <name> = <fire_query>`

You can opt to use built-in [Metrics](/t/)

For example, alarm on a cpu metric that triggers whenever cpu usage exceeds 65% for over 15 seconds in a 30 second interval. The base Op command to configure this alarm looks like this and includes the following parameters:

<op>alarm high_cpu_alarm = (cpu_usage > 65 | sum(30)) >= 15.0</op>

```c
Created alarm 'high_cpu_alarm'.
```

- `Alarm Name` - must be alphanumeric, use underscores and/or dashes, and globally unique
- `fire_query` - this is the condition that triggers the alarm. It must be a valid Op statement.

The above loads an alarm called high_cpu_alarm. But more details are needed (which are added by executing additional Op commands) to fully define and enable the alarm:

</mode>

<mode type="ui">

<youtube id="c3JPXM9qQ6g" />

1. Click on **New Alarm**
2. Enter a **Name** and **Description**
3. Leave the **Enable alarm** box checked
4. Select **Metric** for **Alarm based on**
5. Select the **Resource Type** and apply a filter
6. Select the **Metric** from the drop down
7. Set the **Threshold** value - this can be a whole number or decimal
8. Set the **Raise alarm when** and **Resolve alarm when** values - these are whole numbers
9. Set the **Alarm description** and **Resolve description**
10. Optionally, set **Mute when** to disable the alarm under certain conditions
11. Click **Save**

</mode>

</modes>

## Edit an Alarm

1. From the **Configured alarm list**, click on an alarm
2. Modify the alarm settings
3. Click **Save**

## Delete an Alarm

1. To Disable or Delete, select the alarm by clicking on the checkbox next to the alarm name
2. In the upper right hand corner of the dashboard, click on the dropdown next to **Select**
3. Click **Disable** or **Delete**
4. To delete, you will be prompted to enter the alarm name as a confirmation before deleting it

---

- `clear_query` - what condition clears the alarm?

<op>{'high_cpu_alarm.clear_query = (cpu_usage < 65 | sum(30)) >= 15.0'}</op>

- `resource_query` - what resources is this alarm acting upon (host, pod, or container)?

<op>high_cpu_alarm.resource_query = host resource_type</op>

- `resource_type` - what is the specific resource type (HOST, POD, CONTAINER)?

<op>high_cpu_alarm.resource_type = "HOST"</op>

- `raise_for` - only local alarm is currently supported

<op>high_cpu_alarm.raise_for = "local"</op>

- `raise_family` - to make this alarm interchangeble and actionable between the CLI and UI, add family type of "custom"

<op>high_cpu_alarm.raise_family = "custom"</op>

- `metric_name` - what specific resource metric is being monitored?

<op>high_cpu_alarm.metric_name = "cpu_usage"</op>

- `condition_type` - is the condition defined as above or below the threshold?

<op>high_cpu_alarm.condition_type = "above"</op>

- `condition_value` - what is the condition value?

<op>high_cpu_alarm.condition_value = "65"</op>

Executing these Op commands updates the alarm.

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613944985773/screen-shot-4.png)

By default, all defined objects in Op start in a disabled state.  We want to make sure that we only perform authorized and fully prepared computation.  Op is all about safety and reliability.  With the definition, clear condition, and resources statements, our alarm is ready to be deployed.  To do this, we enable our alarm so that it runs on each of our hosts, constantly checking if cpu usage has exceeded the threshold:

- `enable` - the default is false

<op>enable high_cpu_alarm</op>

To verify that the alarm is defined and enabled, we use the `list` command:

<op>list alarms | name="high_cpu_alarm"</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613946127232/screen-shot-5.png)

Our new alarm is also visible in the Shoreline UI. But it needs more information in order to be synchronized with and editable in the UI. This step is optional but highly so that operators can leverage both the CLI and UI seamlessly.

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613946825229/screen-shot-7.png)

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613946877063/screen-shot-6.png)

The Op commands to synchronize this alarm to the UI are:

- `description`

<op>high_cpu_alarm.description = "cpu_usage exceeds 65% 15s out of 30s"</op>

- `fire_title_template`

<op>high_cpu_alarm.fire_title_template = "cpu usage above threshold"</op>

- `fire_short_template`

<op>high_cpu_alarm.fire_short_template = "cpu usage above threshold"</op>

- `resolve_title_template`

<op>high_cpu_alarm.resolve_title_template = "cpu usage below threshold"</op>

- `resolve_short_template`

<op>high_cpu_alarm.resolve_short_template = "cpu usage below threshold"</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613947868572/screen-shot-8.png)

The alarm is now completely configured to be managed from both the CLI and the UI.

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613948212288/screen-shot-9.png)

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613948223026/screen-shot-10.png)

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613948246205/screen-shot-11.png)

To see all the configured alarms, use the `list` command.

<op>list alarms</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613949109227/screen-12.png)


Overview

Alarms

Alarm Properties


<youtube id="brZ1nvb1y3s" />

## Create an Action

<modes>

<mode type="ui">

1. Click on **New Action**
2. Enter a **Name** and **Description**
3. Leave the **Enable Action** box checked
4. Enter **Start**,  **Completion**, and **Error** titles and descriptions
5. Click **Save**

</mode>

</modes>

## Edit an Action

<modes>

<mode type="ui">

1. From the **Configured Action list**, click on an Action
2. Modify the Action settings
3. Click **Save**

</mode>

</modes>

## Disable an Action

<modes>

<mode type="ui">

1. To Disable or Delete, select the Action by clicking on the checkbox next to the Action name
2. In the upper right-hand corner of the dashboard, click on the dropdown next to **Select**
3. Click **Disable** or **Delete**
4. To delete, you will be prompted to enter the alarm name as a confirmation before deleting it

</mode>

</modes>

## Delete an Action

<modes>

<mode type="ui">

1. To Disable or Delete, select the Action by clicking on the checkbox next to the Action name
2. In the upper right hand corner of the dashboard, click on the dropdown next to **Select**
3. Click **Disable** or **Delete**
4. To delete, you will be prompted to enter the alarm name as a confirmation before deleting it

</mode>

</modes>

---

Thus far, we have learned how [Op](/t/op) discovers [Resources](/t/resource), how to measure how those resources are operating, and how to set up an [Alarm](/t/alarm) when an event is triggered.  [Monitoring](/ui/monitoring) and [alarming](/t/alarm) are crucial activities, but they only tell us what or when something is going wrong.  The problems don't fix themselves.  However, as operators, we are responsible for mitigating and remediating these problems.

Instead of manually performing Actions, [Op](/t/op) lets us automate Actions and encode an entire workflow. Let's create an Action that displays the top 5 most CPU intensive processes and output the PID, command, CPU and memory usage.

The base [Op](/t/op) command to configure this Action includes the following parameters:

- `Action Name` - must be alphanumeric, use underscores and/or dashes, and be globally unique
- `Command` - this is the shell command.

<op>action high_cpu_action = `ps aux`</op>

```c
Created action 'high_cpu_action'.
```

- `resource_query` - this must be a valid [Op](/t/op) statement that takes Action on [Op](/t/op) [Resources](/t/resource).

<op>high_cpu_action.resource_query = host | .pod | .container | app="shoreline" | limit=5</op>

```c
Updated action 'high_cpu_action'.
```

<note type="warning">

The `resource_query` property is required
</note>

[Op](/t/op) is all about safety and reliability. To ensure only authorized and intended Actions are executed all defined objects in [Op](/t/op) start in a disabled state.  Therefore, to enable our Action, use the [enable](/op/commands/enable) command.

<op>enable high_cpu_action</op>

```c
Updated action 'high_cpu_action'.
```

<note type="tip">

The [list action](/op/commands/list#action) command verifies that the Action was defined.

<op>

<span className={'text-lg'}>list action | name="high_cpu_action"</span>
</op>

```c
 TYPE   | NAME            | ENABLED | RESOURCE_QUERY                                           | PARAMS | COMMAND  | TIMEOUT
ACTION | high_cpu_action | 0       | host | .pod | .container | app = "shoreline" | limit = 5 | []     | `ps aux` | 60
```

</note>

> for now let’s just document that resource query is option if action is used in auto remediation and mandatory if used by itself as in the case of many of the op packs

Our new Action is also visible in the Shoreline UI. But it needs more information in order to be synchronized with and editable in the UI. This step is optional but highly so that operators can leverage both the CLI and UI seamlessly.

![action-in-ui](/assets/images/articles/actions/action-in-ui.png)
![action-description-panels](/assets/images/articles/actions/action-description-panels.png)

The [Op](/t/op) commands to synchronize this Action to the UI are:

- `start_title_template`

<op>high_cpu_action.start_title_template = "high cpu remediation has started"</op>

- `start_short_template`

<op>high_cpu_action.start_short_template = "high cpu remediation has started"</op>

- `error_title_template`

<op>high_cpu_action.error_title_template = "high cpu remediation resulted in error"</op>

- `error_short_template`

<op>high_cpu_action.error_short_template = "high cpu remediation resulted in error"</op>

- `complete_title_template`

<op>high_cpu_action.complete_title_template = "high cpu remediation has completed"</op>

- `complete_short_template`

<op>high_cpu_action.complete_short_template = "high cpu remediation has completed"</op>

The Action is now completely configured to be managed from both the CLI and the UI.

![](https://files.helpdocs.io/6m3dgskxth/articles/67uc28d8fp/1613957427212/screen-shot-16.png)

To see all the configured Actions, use the _List_ command.

<op>list actions</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/67uc28d8fp/1613957485129/screen-shot-17.png)


Actions


The final piece in Op automation is connecting alarms in actions.  When we detect a pattern of interest, let's kick off the corresponding mitigation.  Runbooks encode this connection, but they don’t address detection and migration.  Those activities are left as manual tasks for the operator. To truly have end-to-end automation, we create a bot.  Bots bind alarms and actions together using an IF-THIS-THEN-THAT style.

Let's create a bot that upon high cpu, kills the background-logger process.  As we said before, the background logger will restart again later to finish pushing logs, after the cpu usage has come down, so it is safe to disable it temporarily.

<op>bot logger_stopper = if cpu_high then killall("background-logger") fi</op>

Just as we enabled alarms, let's enable our bot:

<op>enable logger_stopper</op>

Now we have entirely completed an automation of what would formerly have been a runbook.  Instead of checking for high cpu on a dashboard, we have an automatic alarm.  Instead of manually killing a background process, we have a defined action. And instead of getting a page, we have a bot that watches the alarm and triggers the mitigating action automatically.


Bots


In the [Resources](/t/resource) article, we learned how to discover resources, filter on them, drill down for more detail, and learn what the relationships are.  But beyond just discovering the resources, we also want to know how our resources are operating.  We want to know host cpu, disk, memory, and network utilization.  We also want to know service latency and error rates.  To get this information, we need metrics. Op metrics are time series data with name, resource, and other tag metadata. Let's get the value for a metric right now.  Let’s get the average cpu utilization for each of our hosts for the last 30 seconds:

<op>host | cpu_usage | window(30s) | resolution=10 | real_time=true | mean(3)</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/0h85lt9lh6/1620105475097/screen-shot-2021-05-03-at-10-17-49-pm.png)

Op resources and metrics naturally mesh together.  Prefixing a metric query with a resource query narrows the metrics returned down to only those metrics associated with the returned resources.

## Creating New Metrics

Like resources, we can also define and save useful metric queries for later.  Let's define the concept of average cpu over an interval:

<op>metric cpu_2_min = cpu_usage | window(120s) | real_time =true</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/0h85lt9lh6/1620105655052/screen-shot-2021-05-03-at-10-20-40-pm.png)

As shown above, our definition statements are fully parameterized.  Op's macro system allows for expression of complex substitution, but with a familiar syntax.

Op also lets users create custom derived metrics. Let's create a new metric called "cpu_usage_new":

<op>metric cpu_usage_new = (100 - 100 \* (metric_query(metric_names="node_cpu_seconds_total", tags={'{'}"mode":"idle"{'}'}) | irate(2) | r_mean)) | lower_bound(0) | upper_bound(100)</op>

If you need to update the formula, simply over-write with the following:

<op>cpu_usage_new.formula = [new formula]</op>

To set unit of measurement (uom) for the metric:

<op>cpu_usage_new.unit = "percent"</op>

Now we can leverage both of the Op commands we have built up.  Let's get the average cpu utilization over the last 30 seconds for each of our hosts:

<op>host | cpu_2_min</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/0h85lt9lh6/1620105793098/screen-shot-2021-05-03-at-10-22-57-pm.png)

If the metric is always measure against a resource type, then you can associate a default resource type with the metric as follows:

<op>cpu_2_min.resource_type = "HOST"</op>

The above examples really show the power of Op.  Multiple layers of substitution allow you to very succinctly express a complex query.  And in the heat of a Sev 1 or similarly critical incident, you will be able to easily remember the commands and execute efficiently.

## List Existing Metrics

Op allows the user to list all previously defined metrics. Imagine you are interactively debugging your cluster using Op, and you want the cpu_2_min metric, defined above, on each of your hosts, but you don’t remember the name of the defined metric. Let’s list our existing metrics.

<op>list metrics</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/0h85lt9lh6/1620105979752/screen-shot-2021-05-03-at-10-26-14-pm.png)

To view just a metric by name:

<op>list metrics | name="cpu_usage_new"</op>

We see that we have an entry in the table for our cpu_2_min metric. We know what this metric represents since we defined it and provided it with an appropriate name. However, what if someone new to the ops team wants more information about the metric? Let’s add a description to our metric.

<op>cpu_2_min.description = "real time cpu usage over last 2 min"</op>

Now, if we list the metrics again:

<op>list metrics</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/0h85lt9lh6/1620106197944/screen-shot-2021-05-03-at-10-29-51-pm.png)

They will now see our metric description alongside the metric name and formula. Op supports standard CRUD (create, read, update, and delete) operations over metric definitions. Please refer to the Op Commands Glossary for more information regarding syntax and supported operations. These features enable an operations team to rapidly build up a shared statement bank of commonly used metrics, which allows for the dissemination of operations knowledge and the ability to quickly gather valuable system information without wasting time reinventing or misremembering commonly used metric formulas.

## Metrics Exporter Support

Op plugs directly into the **[Prometheus](https://prometheus.io/docs/introduction/overview/)** exporter ecosystem. Op can pull metrics from any Prometheus exporter as well as Prometheus itself.  Exporters such as Envoy and cAdvisor are auto-discovered and ingestible by the Shoreline agent.

For further information on Envoy, please see **[Envoy Overview](https://www.envoyproxy.io)**.

For examples of cAdvisor metrics, please see **[Monitoring container metrics using cAdvisor](https://prometheus.io/docs/guides/cadvisor)**.


Metric Sets

Configuration


## Philosophy

Markdown is intended to be as easy-to-read and easy-to-write as is feasible.

Readability, however, is emphasized above all else. A Markdown-formatted
document should be publishable as-is, as plain text, without looking
like it's been marked up with tags or formatting instructions. While
Markdown's syntax has been influenced by several existing text-to-HTML
filters -- including [Setext](http://docutils.sourceforge.net/mirror/setext.html), [atx](http://www.aaronsw.com/2002/atx/), [Textile](http://textism.com/tools/textile/), [reStructuredText](http://docutils.sourceforge.net/rst.html),
[Grutatext](http://www.triptico.com/software/grutatxt.html), and [EtText](http://ettext.taint.org/doc/) -- the single biggest source of
inspiration for Markdown's syntax is the format of plain text email.

## Block Elements

### Paragraphs and Line Breaks

A paragraph is simply one or more consecutive lines of text, separated
by one or more blank lines. (A blank line is any line that looks like a
blank line -- a line containing nothing but spaces or tabs is considered
blank.) Normal paragraphs should not be indented with spaces or tabs.

The implication of the "one or more consecutive lines of text" rule is
that Markdown supports "hard-wrapped" text paragraphs. This differs
significantly from most other text-to-HTML formatters (including Movable
Type's "Convert Line Breaks" option) which translate every line break
character in a paragraph into a `<br />` tag.

When you *do* want to insert a `<br />` break tag using Markdown, you
end a line with two or more spaces, then type return.

### Headers

Markdown supports two styles of headers, [Setext] [1] and [atx] [2].

Optionally, you may "close" atx-style headers. This is purely
cosmetic -- you can use this if you think it looks better. The
closing hashes don't even need to match the number of hashes
used to open the header. (The number of opening hashes
determines the header level.)


### Blockquotes

Markdown uses email-style `>` characters for blockquoting. If you're
familiar with quoting passages of text in an email message, then you
know how to create a blockquote in Markdown. It looks best if you hard
wrap the text and put a `>` before every line:

> This is a blockquote with two paragraphs. Lorem ipsum dolor sit amet,
> consectetuer adipiscing elit. Aliquam hendrerit mi posuere lectus.
> Vestibulum enim wisi, viverra nec, fringilla in, laoreet vitae, risus.
>
> Donec sit amet nisl. Aliquam semper ipsum sit amet velit. Suspendisse
> id sem consectetuer libero luctus adipiscing.

Markdown allows you to be lazy and only put the `>` before the first
line of a hard-wrapped paragraph:

> This is a blockquote with two paragraphs. Lorem ipsum dolor sit amet,
consectetuer adipiscing elit. Aliquam hendrerit mi posuere lectus.
Vestibulum enim wisi, viverra nec, fringilla in, laoreet vitae, risus.

> Donec sit amet nisl. Aliquam semper ipsum sit amet velit. Suspendisse
id sem consectetuer libero luctus adipiscing.

Blockquotes can be nested (i.e. a blockquote-in-a-blockquote) by
adding additional levels of `>`:

> This is the first level of quoting.
>
> > This is nested blockquote.
>
> Back to the first level.

Blockquotes can contain other Markdown elements, including headers, lists,
and code blocks:

> ## This is a header.
>
> 1.   This is the first list item.
> 2.   This is the second list item.
>
> Here's some example code:
>
>     return shell_exec("echo $input | $markdown_script");

Any decent text editor should make email-style quoting easy. For
example, with BBEdit, you can make a selection and choose Increase
Quote Level from the Text menu.


### Lists

Markdown supports ordered (numbered) and unordered (bulleted) lists.

Unordered lists use asterisks, pluses, and hyphens -- interchangably
-- as list markers:

*   Red
*   Green
*   Blue

is equivalent to:

+   Red
+   Green
+   Blue

and:

-   Red
-   Green
-   Blue

Ordered lists use numbers followed by periods:

1.  Bird
2.  McHale
3.  Parish

It's important to note that the actual numbers you use to mark the
list have no effect on the HTML output Markdown produces. The HTML
Markdown produces from the above list is:

If you instead wrote the list in Markdown like this:

1.  Bird
1.  McHale
1.  Parish

or even:

3. Bird
1. McHale
8. Parish

you'd get the exact same HTML output. The point is, if you want to,
you can use ordinal numbers in your ordered Markdown lists, so that
the numbers in your source match the numbers in your published HTML.
But if you want to be lazy, you don't have to.

To make lists look nice, you can wrap items with hanging indents:

*   Lorem ipsum dolor sit amet, consectetuer adipiscing elit.
    Aliquam hendrerit mi posuere lectus. Vestibulum enim wisi,
    viverra nec, fringilla in, laoreet vitae, risus.
*   Donec sit amet nisl. Aliquam semper ipsum sit amet velit.
    Suspendisse id sem consectetuer libero luctus adipiscing.

But if you want to be lazy, you don't have to:

*   Lorem ipsum dolor sit amet, consectetuer adipiscing elit.
    Aliquam hendrerit mi posuere lectus. Vestibulum enim wisi,
    viverra nec, fringilla in, laoreet vitae, risus.
*   Donec sit amet nisl. Aliquam semper ipsum sit amet velit.
    Suspendisse id sem consectetuer libero luctus adipiscing.

List items may consist of multiple paragraphs. Each subsequent
paragraph in a list item must be indented by either 4 spaces
or one tab:

1.  This is a list item with two paragraphs. Lorem ipsum dolor
    sit amet, consectetuer adipiscing elit. Aliquam hendrerit
    mi posuere lectus.

    Vestibulum enim wisi, viverra nec, fringilla in, laoreet
    vitae, risus. Donec sit amet nisl. Aliquam semper ipsum
    sit amet velit.

2.  Suspendisse id sem consectetuer libero luctus adipiscing.

It looks nice if you indent every line of the subsequent
paragraphs, but here again, Markdown will allow you to be
lazy:

*   This is a list item with two paragraphs.

    This is the second paragraph in the list item. You're
    only required to indent the first line. Lorem ipsum dolor
    sit amet, consectetuer adipiscing elit.

*   Another item in the same list.

To put a blockquote within a list item, the blockquote's `>`
delimiters need to be indented:

*   A list item with a blockquote:

    > This is a blockquote
    > inside a list item.

To put a code block within a list item, the code block needs
to be indented *twice* -- 8 spaces or two tabs:

*   A list item with a code block:

        <code goes here>

### Code Blocks

Pre-formatted code blocks are used for writing about programming or
markup source code. Rather than forming normal paragraphs, the lines
of a code block are interpreted literally. Markdown wraps a code block
in both `<pre>` and `<code>` tags.

To produce a code block in Markdown, simply indent every line of the
block by at least 4 spaces or 1 tab.

This is a normal paragraph:

    This is a code block.

Here is an example of AppleScript:

    tell application "Foo"
        beep
    end tell

A code block continues until it reaches a line that is not indented
(or the end of the article).

Within a code block, ampersands (`&`) and angle brackets (`<` and `>`)
are automatically converted into HTML entities. This makes it very
easy to include example HTML source code using Markdown -- just paste
it and indent it, and Markdown will handle the hassle of encoding the
ampersands and angle brackets. For example, this:

    <div class="footer">
        &copy; 2004 Foo Corporation
    </div>

Regular Markdown syntax is not processed within code blocks. E.g.,
asterisks are just literal asterisks within a code block. This means
it's also easy to use Markdown to write about Markdown's own syntax.

```
tell application "Foo"
    beep
end tell
```

```ts
import {
  ArticleInterface,
  ArticleJsonInterface
} from '@/model/article/ArticleInterface';

type ArticleParams = {
  content: string;
  date?: string;
  excerpt?: string;
  section?: string;
  slug?: string;
  title: string;
};

class Article implements ArticleInterface {
  private _content: string = '';
  private _date?: string;
  private _excerpt?: string;
  private _section?: string;
  private _slug?: string;
  private _title: string = '';

  constructor(params?: ArticleParams) {
    if (params) {
      this.content = params.content;
      this.date = params.date;
      this.excerpt = params.excerpt;
      this.section = params.section;
      this.slug = params.slug;
      this.title = params.title;
    }
  }

  get content(): string {
    return this._content;
  }

  set content(value: string) {
    this._content = value;
  }

  get date(): string | undefined {
    return this._date;
  }

  set date(value: string | undefined) {
    this._date = value;
  }

  get excerpt(): string | undefined {
    return this._excerpt;
  }

  set excerpt(value: string | undefined) {
    this._excerpt = value;
  }

  get path(): string {
    return (this.section ? this.section + '/' : '') + this.slug;
  }

  get section(): string | undefined {
    return this._section;
  }

  set section(value: string | undefined) {
    this._section = value;
  }

  get slug(): string | undefined {
    return this._slug;
  }

  set slug(value: string | undefined) {
    this._slug = value;
  }

  get title(): string {
    return this._title;
  }

  set title(value: string) {
    this._title = value;
  }

  toJson(): ArticleJsonInterface {
    return {
      content: this.content,
      date: this.date ?? null,
      excerpt: this.excerpt ?? null,
      path: this.path,
      section: this.section ?? null,
      slug: this.slug ?? null,
      title: this.title
    };
  }
}

export default Article;
```

## Span Elements

### Links

Markdown supports two style of links: *inline* and *reference*.

In both styles, the link text is delimited by [square brackets].

To create an inline link, use a set of regular parentheses immediately
after the link text's closing square bracket. Inside the parentheses,
put the URL where you want the link to point, along with an *optional*
title for the link, surrounded in quotes. For example:

This is [an example](http://example.com/) inline link.

[This link](http://example.net/) has no title attribute.

### Emphasis

Markdown treats asterisks (`*`) and underscores (`_`) as indicators of
emphasis. Text wrapped with one `*` or `_` will be wrapped with an
HTML `<em>` tag; double `*`'s or `_`'s will be wrapped with an HTML
`<strong>` tag. E.g., this input:

*single asterisks*

_single underscores_

**double asterisks**

__double underscores__

### Code

To indicate a span of code, wrap it with backtick quotes (`` ` ``).
Unlike a pre-formatted code block, a code span indicates code within a
normal paragraph. For example:

Use the `printf()` function.


Dimensions

Derived Metrics


As operators, you perform many manual tasks to maintain, monitor, and troubleshoot applications in the production environment.  If you are lucky, these tasks are part of scheduled maintenance operations: a predetermined time block is published, users are notified, systems are quiesced, you go in and do your thing and get out, bring the system back up and inform everyone.

However, more often than not you have to do these manual tasks on-demand to handle a page during an on-call rotation.  Your organization may pursue the goal of NoOps (the idea that a software environment can be completely automated so that there's no need for an operations team to manage it) but we all know that fixing root causes in software takes weeks to months.  And during that time, we keep incurring and executing manual tasks to mitigate the issue.

At the same time, the number of supported environments, Clouds, and software deployments continues to increase, only further putting pressure on your ability to meet SLAs.

Shoreline Op is a purpose-built operations oriented language designed to allow operators and admins to rapidly:

- understand, debug, and fix systems during an operational event
- automate the tasks performed during mitigation in order to reduce or even end future manual processing

## How Op Works

While you (the operator) are debugging, the Op interactive CLI allows you to gather and correlate information about resources, metrics, and system state to give you greater insight and inform the actions you need to take to mitigate the issue.  Afterwards, Op allows the operator to encode behaviors into Actions, Alarms, and Bots to automatically search for and fix the issues, without operator manual intervention going forward.

Using the Op language for "in the moment" debugging and permanent fixing significantly reduces the time to automate.  The statements used during operational work to identify and remediate an issue can be immediately deployed for permanent issue detection and fix.

## The Op Language

Op uses familiar Bash syntax which makes it easy to learn. Its core primitives allow the operator to focus on encoding how to detect and mitigate an event rather than solving distributed systems problems.  Operators tell Op what to look for and how to fix it and then Op handles the undifferentiated heavy lifting such as propagation, error handling, auditing, system lease,...etc.

In the Op Language Guide articles, we will go through how to use Op for both interactive debugging and permanent auto remediation.  We've also published a Glossary that has full documentation for each of the Op commands.



In this 15 minute tutorial, we'll cover the fundamental building blocks of Shoreline. This tutorial will focus on **Op**: Shoreline's domain-specific language for operations. Op allows operators (SREs, DevOps, SysAdmin, and SWE) to:

1.  Interactively debug their systems
2.  Create automated remediations to detect and mitigate issues without operator involvement

This tutorial will focus on Shoreline running in a Kubernetes environment on AWS. As of writing, Shoreline also runs on AWS VMs directly, however, that will not be covered.

### Resources

Operators care about the resources they are responsible for. To that end, Shoreline discovers the hosts, pods, and containers in your environment. Let's begin with a basic resource query to see all of our hosts.

<op>host</op>

Not only does Shoreline discover your resources, it captures their metadata as well. Let's see all of fields that we have captured.

<op>columns host</op>

That's quite a few fields: shoreline captures metadata - tags from ec2 as well as Kubernetes - and makes it all available in one place. Let's turn on the az tag and then see the availability zone of each of our hosts.

<op>config show resources.fields.az</op>
<op>host</op>

We can filter on tags as well. Let's fetch only those hosts in the us-west-2c availability zone.

<op>host | az="us-west-2c"</op>

The pipe operator is inspired by shell's pipe - you use it chain together statements in the Op, passing data from one to the next. In the above, a tag filter placed after a resource query filters down the hosts to only those whose tags match.

Let's combine more statements. Op also discovers resource topology e.g. which pods are scheduled to which hosts and which containers sit in those pods. Let's fetch all of the pods scheduled to those hosts in zone us-west-2c:

<op>host | az="us-west-2c" | .pod</op>

In shell, we often create aliases for commands that we frequently run. Inspired by this, Op lets you define stored statements. Let's create a stored resource query to fetch all the hosts in us-west-2c.

<op>resource uswest2c = host | az="us-west-2c" </op>
<op>uswest2c</op>

Running the above, the uswest2c symbol now resolves to all hosts in that zone. We can also attach metadata to these symbols and list all symbols stored in the collective memory.

<op>uswest2c.description = "Hosts in us-west-2c" </op>
<op>list resources</op>

These symbols are accessible to all operators. This prevents the issue of commands being trapped in folks' heads or growing out of date on wikis.

### Metrics

Operators care about which resources they are responsible for and how these resources are performing. To that end, metrics are integrated into Shoreline as a first class entity. We can combine together resource and metric queries using the pipe operator. Let's fetch the cpu usage for all of our hosts:

<op>host | cpu_usage</op>

Like Prometheus, Shoreline polls Kubernetes exporters to ingest metrics making it easy to place into a Kubernetes environment. Shoreline also supports deploying to virtual machines, and in that environment, Shoreline installs the same exporters to gather metrics.

Op contains an entire metrics expression language. The entire syntax is outside of the scope of this tutorial, but we can do some simple manipulations of our cpu metric. Let's grab it for the last 30 seconds:

<op>host | cpu_usage | window(30s)</op>

Now let's take the average for each resource:

<op>host | cpu_usage | window(30s) | mean(30)</op>

Op also supports dynamic filter. For example, we can filter all our hosts whose cpu_usage is currently greater than 10%.

<op>host | filter(cpu_usage > 10)</op>

Try raising the threshold higher and higher. You'll see more and more hosts filtered out.

<op>host | filter(cpu_usage > 90)</op>

Like resource query, we can also see the definitions of each of our stored metrics. This is a pretty large list - shoreline comes preloaded with common metric queries for cAdvisor and node exporters. We can filter these by name, matching on regex.

<op>list metrics </op>
<op>list metrics | name=~"container"</op>

### Actions

While Shoreline is useful as a resource inventory and observability tool, it really shines as a mitigation tool i.e. shoreline lets you take action. To that end, Shoreline supports executing distributed commands. Distributed commands can run on hosts or on containers. Let's start off by creating a resource query that returns our shoreline agent containers: we'll use these to test actions.

<op>resource shoreline = host | .pod | app="shoreline" | .container </op>
<op>shoreline</op>

Actions fully support shell: they are just snippets of shell. These snippets can even reference entire scripts. Let's run an action in each of our shoreline containers to order the processes descending by memory. Then we will limit those to just the top 10:

<op>shoreline | \`top -b -o -%MEM -n 1 | tail -n 10\`</op>

Like resources and metrics, we can also store actions as well. To do this, we define an action.

<op>action top_memory = \`top -b -o -%MEM -n 1 | tail -n 10\`</op>

We can now use it by name:

<op>shoreline | top_memory</op>

Again, like resources and metrics, we can attach metadata to our actions and list all of the stored actions:

<op>top_memory.description = "Top processes by memory usage" </op>
<op>list actions</op>

### Alarms

Individual resource, metric, and action are useful for interactive debugging, but the power to perform automated remediation comes from combining them together into higher constructs. The first of these is an alarm. Alarms can fire on metrics and/or Linux commands. This means you can create alarms that take into account fast moving metrics along with system state.

Let's create the most basic type of alarm, an alarm on a single metric. Let's define the alarm to fire when the cpu_usage for a host has exceeded 70%, 20 times in the last 30 seconds.

<op>alarm high_cpu_alarm = ((cpu_usage > 70) | sum(30)) >= 20</op>

Next, we need to define how to clear the alarm. Alarm clear and fire might not be symmetrical e.g. we might want it to be easier to enter alarm than to exit alarm so that we have some certainty that the alarm won't flap up and down.

In this case though, we'll define a symmetrical alarm, on just the opposite condition i.e. we'll clear the alarm when cpu usage is greater than 70% less than 20 times in the last 30 seconds.

<op>{'high_cpu_alarm.clear_query = ((cpu_usage > 70) | sum(30)) < 20'}</op>

Next, we need to instruct the alarm to monitor a set of our resources. Let's monitor all of the hosts by setting the alarm's resource query to host:

<op>high_cpu_alarm.resource_query = host</op>

Like the other objects that we discussed before, we can also attach metadata to our alarm:

<op>high_cpu_alarm.description = "Alarm when host cpu usage is high"</op>

Our alarm is now fully defined, but it is not yet running. To run the alarm, constantly checking to see if the condition is met, we need to enable it. There is a special command for this called enable:

<op>enable high_cpu_alarm</op>

Finally, like all other objects, we can list alarms to see what we have created:

<op>list alarms</op>

### Bots

So far we have created an alarm, but nothing happens if the alarm fires. To solve that problem, we need to link an action to the alarm. To do that, we'll define a bot. Bots are if-this-then-that style structures that link alarms to actions. Let's create a bot that runs our top_memory action from before whenever the high_cpu_alarm fires:

<op>bot check_memory_bot = if high_cpu_alarm then top_memory fi</op>

As usual, we'll add a description to the bot:

<op>check_memory_bot.description = "If host has high cpu, check shoreline container memory"</op>

When the alarm fires, now the top_memory action will be executed. But where will it be executed? By default, Shoreline will try to execute the action on the same resource that triggered the alarm, in this case a host. Let's override that to execute the top_memory action in the shoreline container scheduled on the host that had the alarm.

<op>top_memory.resource_query = shoreline</op>

When an action is used in an alarm, we need to enable the action as well. This tells the system that it is safe to execute the action on your behalf. We'll also need to enable the bot so that it links the alarm and action:

<op>enable top_memory </op>
<op>enable check_memory_bot</op>

Finally, let's inspect our bot definitions before finishing the tutorial:

<op>list bots</op>

### Conclusion

We hope that you found this to be a useful introduction to using Op and Shoreline. There are many more capabilities and parameters that were not covered in this tutorial. Those can be found throughout the rest of the documentation or by reaching out via slack or email. But, we hope that this gives you enough starting knowledge to begin automating operational tasks!


Tutorial


Op is a powerful interactive CLI based upon an operations oriented language for gathering information about your infrastructure and proactively managing issues in order to maintain a stable operational environment.  Op has five types of objects that you will be working with:

1. Resources - the infrastructure objects in your environment.  Resources can be hosts, pods, containers, virtual machines, database instances etc.  On different platforms, you can have different resources e.g. pods and containers for Kubernetes and virtual machines for AWS, GCP, or Azure.
2. Metrics - time series data associated with your resources such as CPU utilization, latency, throughput, or error rate.  You can query the metrics gathered to monitor the environment.  You also use metrics to determine if a condition or threshold has been met which requires an Action to be taken.
3. Alarms - indicators that tell you when something has gone wrong and/or needs attention.  Alarms are defined by a combination of resources, metrics, Linux commands, and system state and are triggered when a condition is met.
4. Actions - shell commands and shell scripts that help you mitigate and remediate the problem that an Alarm has identified.  The Op CLI comes with a basic set of actions.  You can define custom actions and encode an entire operational workflow into a runbook using Actions.
5. Bots - bind Alarms and Actions together using IF-THEN-ELSE constructs.  Bots specify the action or set of actions to take when the alarm triggers.

Op has five types of functions that you will be performing on objects:

Op functions are similar to the basic **C**reate **R**ead **U**pdate **D**elete (CRUD) functions that you are likely used to working with in other languages.

1. Define - creates new objects.
2. Get - returns the value of a specified field inside of an object.
3. List - returns all instances of the object type specified.
4. Update - modifies the description, value, arguments, approvers, and enabled state of existing objects.
5. Delete - erases existing objects.

Op objects support symbolic manipulation.  The Op functions bind the symbol to the underlying query or action.  This Glossary provides the syntax and examples for each function.


Introduction

Op Commands


Op has five core types of objects:

- **Resources** - The infrastructure objects in your environment.
- **Alarms** - Checks, built on metrics and shell commands, that determine if the state of your resources is in issue.
- **Bots** - Bind alarms and actions together: when an alarm fires a mitigating action is automatically run to fix the issue
- **Actions** - Shell commands and shell scripts that act on your resources.
- **Metrics** - Time series data data associated with your resources such as CPU utilization, latency, throughput, or error rate.

We will start our Op walkthrough with Resources; each of these objects will be explained in detail in separate sections.

## Introduction to Resources

Resources can be hosts, pods, containers, virtual machines, database instances etc. On different platforms, we can have different resources e.g. pods and containers for Kubernetes and virtual machines for AWS, GCP, or Azure. Op explicitly models resources and their relationships. Let's start off with a simple resource query from the command line:

<op>host</op>

```
 ID | TYPE | NAME                | REGION    | AZ
 1  | HOST | i-064246cabcbb73067 | us-east-1 | us-east-1b
 2  | HOST | i-0812b3fd801632ecc | us-east-1 | us-east-1b
 3  | HOST | i-0189e804e46578b34 | us-east-1 | us-east-1a
```

Above you can see each of the host resources that appear in your environment. In this example, we have 3 AWS EC2 instances, all in the us-west-2 region, different availability zones. Each of these hosts is running a Shoreline agent. The agent is a program which communicates to the Shoreline backend, sends up telemetry and other environmental data, and receives various commands to carry out. In the above output, you can see each resource's name, region, and zone. Every object in Shoreline has a globally unique id and a human readable name.

Objects also have many other fields and you can configure their output. For example, hosts have tags that encode information such as geography or workload. We can interactively adjust this output for the session.

In this example, we have resources in AWS that have tags. Tags are name-value pairs that are used to categorize and organize resources. Let's print out the az tag (the availability zone tag, currently called “topology.kubernetes.io/zone”) for each host.

The resources_tags() function tells us what kind of tags our resources have. For example, running resources_tags(type=”HOST”) shows that we have a region and a zone tag for our hosts (among other tags). We can use this function to find existing tags and to better filter the resources we’re looking for.

<note>Currently, the CLI only outputs the JSON response for this function. Formatting the output is planned for a future release of Op.</note>

So how can you find a particular tag and filter by it and display the column in the CLI?

<op>config hide resources.fields.sl_TestLabel</op>

To change it permanently you can add it to the config (`.ops.yaml`), e.g.:

```yaml
table:
  sections:
    resources:
      show:
      - fields.kubernetes.io.arch
      hide:
      - fields.sl_TestLabel
```

<note>Slash characters (`/`) are converted to dots (`.`). There is also a provision for shortening resource tags (e.g. `kubernetes.io` -> `kube`).</note>

## Resource Definitions

Let's look at the alarms, actions and bots associated with your resources. To do this, we use the list command.  
<op>list alarm</op>

![](/assets/images/articles/v1/op-language/chapter-3-screen-shot-2.png)

<op>list action</op>

![](/assets/images/articles/v1/op-language/chapter-3-screen-3.png)

<op>list bot</op>

![](/assets/images/articles/v1/op-language/chapter-3-screen-shot-4.png)

For each of the above commands, we see the default output columns: name of the definition, description, and the formula. For alarms, we also see the more complex structure including resources to run the alarms on, and the clear query to resolve the alarm. The names of each of these objects is the name that we can refer to the object by. These names are globally unique.  
Above we ran a resource query to actually query for our resources. We can also get the definitions of our resource and metric queries, just like we do for alarms, actions, and bots:

<op>list resource</op>

```c
 TYPE     | NAME      | FORMULA
 RESOURCE | host      | resources(type="HOST")
 RESOURCE | pod       | resources(type="POD")
 RESOURCE | container | resources(type="CONTAINER")
```

<op>list metric</op>

| TYPE   | NAME                        | DESCRIPTION    | FORMULA                                                               |
| ------ | --------------------------- | -------------- | --------------------------------------------------------------------- |
| METRIC | cpu_usage                   | Host cpu usage | (100 - 100 \* (metric_query(metric_names="node_cpu_seconds_total",... |
| METRIC | cpu_usage_pre               | test           | metric_query(metric_names="cpu_usage_pre")                            |
| METRIC | disk_available              |                | metric_query(metric_names="node_filesystem_avail_bytes")              |
| METRIC | network_throughput          |                | metric_query(metric_names="node_network_transmit_packets_total")      |
| METRIC | packet_loss                 |                | metric_query(metric_names="node_network_transmit_drop_total")         |
| METRIC | os_procs_running            |                | metric_query(metric_names="node_procs_running")                       |
| METRIC | os_procs_blocked            |                | metric_query(metric_names="node_procs_blocked")                       |
| METRIC | network_transmit_packets_v2 |                | metric_query(metric_names="node_network_transmit_packets_total")      |
| METRIC | network_transmit_bytes_v2   |                | metric_query(metric_names="node_network_transmit_bytes_total")        |
| METRIC | network_receive_bytes_v2    |                | metric_query(metric_names="node_network_receive_bytes_total")         |

This command lists the metric, its description, and underlying formula. As you can see in this example, Op allows you to define custom and complex metrics, specific to your environment.

Metric and resource definitions both support symbolic manipulation. As you can see above, the definition of host is actually resources(type="HOST"). What this means is that when you typed in host above, the host symbol was translated to that resource query. We will do much more of this later. Op fully supports user defined substitutions.

## The Op Pipe Operator

The next major language feature of Op is the pipe '|' operator. The pipe operator looks just the commonly used operator from the Bash shell and it behaves in a similar way. For example, let's filter our hosts down to only those in the availability zone us-west-2b.  
Like the conventional pipe operator from Bash, the pipe operator allows us to define processing pipelines that filter our results as they are passed through. But this pipe operator has some other features as well. Let's combine together two different type expressions. Let's get all our hosts in us-west-2b and then get their current CPU utilization for the last 5 seconds:

<op>host | az = "us-west-2b" | cpu_usage | window(5s) | real_time=true</op>

```
 ID | TYPE | NAME                | REGION    | AZ         | TIMESTAMPS          | CPU_USAGE
 1  | HOST | i-064246cabcbb73067 | us-east-1 | us-east-1b | 2021/06/01 12:56:18 |      8.00
    |      |                     |           |            | 2021/06/01 12:56:19 |      8.50
    |      |                     |           |            | 2021/06/01 12:56:20 |      7.50
    |      |                     |           |            | 2021/06/01 12:56:21 |      8.50
 2  | HOST | i-0812b3fd801632ecc | us-east-1 | us-east-1b | 2021/06/01 12:56:18 |      5.50
    |      |                     |           |            | 2021/06/01 12:56:19 |      4.50
    |      |                     |           |            | 2021/06/01 12:56:20 |      6.75
    |      |                     |           |            | 2021/06/01 12:56:21 |      4.75
```

Op allows us to bring together different types of objects into a pipeline. Specifically, Op pipelines support everything an operator needs in order to do interactive debugging: resources, metrics, and shell commands. For example, to get all of the pods, the containers scheduled to the host, and what they are running for each host running over 80% cpu utilization, we would use the following command with 'ps':

<op>host | cpu_usage | window(1s) > 0.8 | .pod | .container | `ps aux`</op>

<vc version="next">

How did Op do this? We would use the explain command to see what Op did:

<op>explain host | cpu_usage | window(1s) > 0.8 | .pod | .container | `ps aux`</op>

Like SQL, _explain_ shows what Op would do, but doesn't actually run the statement. Instead, it shows which steps the plan would run, on which hosts, and it what order. A step is a single piece of work that is performed. Steps are dependent on each other; hence, they enforce an order of execution. Op handles the process of determining the appropriate plan for your statement by decomposing it into steps to carry out what you define. Each step has a sequence number. Each step depends on the execution of certain sequence numbers. As the steps execute, an Op object called a Scope is built up i.e. the scope is a summary of all the computation performed by the statement.

Here are the steps for the above plan:

- The backend service runs the resource query step to determine which hosts are in us-west-2a.
- A metric query step is run on each of these hosts to fetch the live CPU usage.
- For each host where the metric query exceeds 80%, a local resource query step is run to obtain the pods and containers. Finally, a Linux command step "ps aux" is run in each of these containers.

</vc>



The `enable` command enables [Actions](/t/action), [Alarms](/t/alarm), and [Bots](/t/bot).

<note type="warning">

By default, all potentially dangerous objects are disabled.
</note>

## Define resource

Syntax: `resource <resource_name> = resources(type="xxx")` where "xxx" is a resource type such as container, pod. VMs...etc.

Examples:

<op>resource host = resources(type="HOST")</op>
<op>resource pod = resources(type="POD")</op>
<op>resource aws = resources() | cloud_provider="aws"</op>

"Host" is case sensitive

## Define metric

Syntax: `metric <metric_name> = <expression>`

Examples:

<op>metric cpu*usage = 100 - 100 * (metric*query(metric_names="node_cpu_seconds_total", tags={'{'}"mode":"idle"{'}'}) | irate(2) | r_mean)</op>
<op>metric pod_reqs_250m_percentile = 100 * pod_reqs_250m_bucket / pod_reqs_count</op>

## Define alarm

Syntax: `alarm <alarm_name> = <condition>`

Examples:

<op>alarm cpu_alarm = cpu_usage > 90</op>
<op>alarm high_disk = pod_disk_usage > 60</op>

After defining the alarm, you need to specify the resource query. In the above examples, you would specify the resources for cpu_alarm with op> cpu_alarm.resource_query=host -and - you would specify the high_disk resource afterwards with op> high_disk.resource_query=pod

## Define action

Syntax: `action <action_name> = <shell command>`

Examples:

<op>action temp_files = `find /tmp -size +100M`</op>
<op>action drop_page_cache = `sync; echo 1 > /proc/sys/vm/drop_caches`</op>

After defining the action, you need to specify the resource query. In the above examples, you would specify the resources for temp_files with `op> temp_files.resource_query=host` and you would specify the drop_page_cache resource afterwards with `op> drop_page_cache.resource_query=host | .pod`

## Define bot

Syntax: `bot <bot_name> = if <alarm_name> then <action_name> fi`

Example:

<op>bot cache_free = if mem_alarm then drop_page_cache fi</op>


Enable

Enable Command


## Define resource

Syntax: `resource <resource_name> = resources(type="xxx")` where "xxx" is a resource type such as container, pod. VMs...etc.

Examples:

<op>resource host = resources(type="HOST")</op>
<op>resource pod = resources(type="POD")</op>
<op>resource aws = resources() | cloud_provider="aws"</op>

"Host" is case sensitive

## Define metric

Syntax: `metric <metric_name> = <expression>`

Examples:

<op>metric cpu*usage = 100 - 100 * (metric*query(metric_names="node_cpu_seconds_total", tags={'{'}"mode":"idle"{'}'}) | irate(2) | r_mean)</op>
<op>metric pod_reqs_250m_percentile = 100 * pod_reqs_250m_bucket / pod_reqs_count</op>

## Define alarm

Syntax: `alarm <alarm_name> = <condition>`

Examples:

<op>alarm cpu_alarm = cpu_usage > 90</op>
<op>alarm high_disk = pod_disk_usage > 60</op>

After defining the alarm, you need to specify the resource query. In the above examples, you would specify the resources for cpu_alarm with op> cpu_alarm.resource_query=host -and - you would specify the high_disk resource afterwards with op> high_disk.resource_query=pod

## Define action

Syntax: `action <action_name> = <shell command>`

Examples:

<op>action temp_files = `find /tmp -size +100M`</op>
<op>action drop_page_cache = `sync; echo 1 > /proc/sys/vm/drop_caches`</op>

After defining the action, you need to specify the resource query. In the above examples, you would specify the resources for temp_files with `op> temp_files.resource_query=host` and you would specify the drop_page_cache resource afterwards with `op> drop_page_cache.resource_query=host | .pod`

## Define bot

Syntax: `bot <bot_name> = if <alarm_name> then <action_name> fi`

Example:

<op>bot cache_free = if mem_alarm then drop_page_cache fi</op>


Define

Define Function


### Get alarm

Returns the value of `field_name` inside of `alarm_name` which indicates the resource the alarm is monitoring.

Syntax: `<alarm_name.field_name>`

<op>cpu_alarm.fire_query</op>

### Get action

Returns the value of `field_name` inside of `action_name` which indicates the resource the action is monitoring.

Syntax: `<action_name.field_name>`

<op>cpu_info.command</op>

### Get bot

Returns the value of `field_name` inside of `bot_name` which indicates the alarm, action, or description that the bot is binding.

Syntax: `<bot_name.field_name>`

Examples:

<op>cpu_bot.alarm</op>
<op>cpu_bot.action</op>
<op>cpu_bot.description</op>


Get Function


### Delete resource

Syntax: `delete <resource_name>`

<op>delete host</op>
<op>delete cpu_bot</op>

### Delete metric

Syntax: `delete <metric_name>`

<op>delete cpu_usage</op>
<op>delete mem_usage</op>

### Delete alarm

Syntax: `delete <alarm_name>`

<op>delete cpu_high</op>
<op>high_latency_and_cpu</op>

### Delete action

Syntax: `delete <action_name>`

<op>delete large_tmp_files</op>
<op>delete kill_container</op>

### Delete bot

Syntax: `delete <bot_name>`

<op>delete logger_stopper</op>
<op>delete kill_background_jobs</op>


Delete

Delete Function


### Update resource

Syntax:

- `<resource_name>.description = "<a description>"`
- `<resource_name>.value> = "<a value>"`

Examples:

<op>host.description = "this command lists all hosts"</op>
<op>aws.description = "this command lists all resources on AWS"</op>

### Update metric

Syntax:

- `<metric_name>.description = "<description>"`
- `<metric_name>.formula = "<formula>"`

Examples:

<op>cpu_usage.description = "this metric lists cpu usage"</op>
<op>cpu_usage.formula = "100 - 100 \* (metric_query(metric_names="node_cpu_seconds_total", tags={'{'}"mode":"idle"{'}'}) | irate(2) | r_mean"</op>

### Update alarm

Before you can update an alarm, you will need to create the alarm, specify the resource, and enable the alarm.  By default, alarms are disabled.

Here are the values that you can update for an alarm:

- `fire_query`
- `metric_name`
- `resource_type`
- `private`
- `resolve`
- `resource_query`
- `resource_tags`
- `clear_query`
- `condition_type`
- `mute_query`
- `user_id`
- `family`
- `condition_value`
- `enabled`
- `visibility`
- `raise_for`

Syntax:

- `<alarm_name>.description = "<a description>"`
- `<alarm_name>.fire_query> = "<new query>"`
- `enable <alarm_name>`
- `disable <alarm_name>`

Examples:

<op>cpu_alarm.fire_query = (metric_query(metric_name="cpu") | window(10s)) > 85</op>
<op>cpu_alarm.description = "a description"</op>
<op>mem_alarm.resource_type = "container"</op>
<op>enable cpu_alarm</op>
<op>disable cpu_alarm</op>

### Update action

Before you can update an action, you will need to create the action, specify the resource, and enable the action.  By default, actions are disabled.

Syntax:

- `<action_name>.description = "<a description>"`
- `enable <action_name>`
- `disable <action_name>`

Examples:

<op>cpu_info.command = `cat /proc/cpuinfo`</op>
<op>enable cpu_info</op>
<op>disable cpu_info</op>

### Update bot

Syntax:

- `<bot_name>.description = "<a description>"`

Examples:

<op>cpu_bot.action = cpu_info</op>
<op>enable cpu_bot</op>


Update

Update Function


Dynamic filter allows user to filter resources in Op by the result of a query at execution time, rather than at planning time. The general syntax is `resource_query | filter(criteria)`, which from a user perspective results in a subset resource query containing only the resources that satisfy the criteria. The supported criterias are metrics queries, Linux commands, and/or annotation queries.

### Filtering by Metric Queries

When filtering by a metric query with multiple values, we retain a resource if any metric value satisfies the criteria. Since metric queries support and/or conditions and nested query trees, dynamic filter also supports such metric queries. If no comparison value is specified, it defaults to !=0 (nonzero data).

Retrieve hosts with nonzero cpu_usage value.

<op>host | filter(cpu_usage)</op>

Retrieve hosts with cpu_usage greater than 40 and mem_used greater than 95.

<op>host | filter((cpu_usage > 40 ) and (mem_used > 95))</op>

When multiple data points exist in a metric query, the filter criteria is satisfied if any of the data points satify the condition. If no data point exists in a metric query, then by default that resource does not satisfy the filter criteria and will be excluded by the filter. However, you can change the default behavior by setting the parameter `include_missing=true`, which will cause resources without data to be included by the filter.

<op>host | .pod | filter(pod_memory_usage) | include_missing=true</op>

### Filtering by Linux Commands

Filtering by Linux commands is evaluated based on the exit codes. Op supports integer exit code comparisons using `==, !=, >, >=, <, <=`. If no comparison is specified, it defaults to `== 0` (i.e. the Linux command succeeded). Op does not currently support and/or comparisons or nested trees of Linux commands, but users can use the Linuc and/or primitives to achieve this behavior. Op also does not currently support comparisons on stdouts, but users can use the Linux command `exit` or built-in Linux comparators to achieve this behavior.

Filter hosts where \`test -f myfile\` returns exit code 0 or command executed successfully.

<op>host | filter(`test -f myfile`)</op>

Filter hosts where \`test -f myfile\` returns a nonzero exit code.

<op>host | filter(`test -f myfile` != 0)</op>

Filter hosts where \`test -f myfile\` returns an exit code of at least 5.

<op>host | filter(`test -f myfile` >= 5)</op>

Filter hosts where either command returns exit code 0.

<op>host | filter(`test -f myfile || test -f myfile2`)</op>

### Filtering by Annotation Queries

When filtering by an annotation query that ends in `count` , we can specify a comparison to compare the number of annotations. If `count` is omitted, the comparison defaults to `count > 0` , i.e. we retain resources where annotations are present.

Filter hosts where my_alarm has fired.

<op>host | filter((annotations | type="ALARM" | name="my_alarm"))</op>

Filter hosts where my_alarm has fired more than twice.

<op>host | filter((annotations | type="ALARM" | name="my_alarm" | count) > 2)</op>

Filter hosts where an action is not completed.

<op>host | filter((annotations | type="ACTION" | status != "completed")</op>

### Taking Action on Filtered Set

Op can follow a filter with a metric query, Linux command, or annotation query, which will execute only on the set of resources resulting from the filter. Below are some examples:

<op>host | filter(cpu_usage > 40) | memory_usage</op>

<op>host | filter(cpu_usage > 40) | `ls` </op>

<op>host | filter(cpu_usage > 40) | annotations</op>

### Chaining Filters and Filtering at Different Levels

We can filter out any level of resources (host, pod, or container). Additionally, filters can be sequenced, where later filters will further refine the set of resources resulting from the previous filters.

Filter containers where \`ls\` succeeds (i.e. containers that support \`bash\`).

<op>host | .pod | .container | filter(`ls`)</op>

Filter hosts with high cpu_usage that also have \`myfile\` present.

<op>host | filter(cpu_usage > 40) | filter(`test -f myfile`)</op>

Op also supports filter in the middle of a resource query to only filter at that level. For example,

Run \`ls\` on containers whose hosts have cpu_usage above 40.

<op>host | filter(cpu_usage > 40) | .pod | .container | `ls` </op>

Filter containers where \`ls\` succeeds whose hosts have cpu_usage above 40.

<op>host | filter(cpu_usage > 40) | .pod | .container | filter(`ls`)</op>


Dynamic Filters


The `list` command returns all instances of the object type specified - [Actions](/t/action), [Alarms](/t/alarm), [Bots](/t/bot), [Metrics](/t/metric), [Resources](/t/resource), or all objects. The object can be further qualified by piping (adding) an attribute name.

Syntax: `list <object_name> | <filter>`

## List Functions

Below are the `list` functions that [Op](/t/op) supports.

### action

`list action` lists all [Actions](/t/action).

<op>list action</op>

```

```

### alarm

`list alarm` lists all [Alarms](/t/alarm).

<op>list alarm</op>

```

```

### all

`list all` lists all objects.

<op>list all</op>

```
  TYPE       | NAME            | ENABLED | FIRE_QUERY                                | DESCRIPTION
  RESOURCE   | app_containers  | ---     | ---                                       |
  ALARM      | high_cpu_alarm  | 0       | ( cpu_usage > 65 | sum ( 30 ) ) >= 15 . 0 |
  RESOURCE   | host            | ---     | ---                                       |
  METRIC     | cpu_usage       | ---     | ---                                       |
  METRIC     | cpu_usage_pre   | ---     | ---                                       | test
```

### bot

`list bot` lists all [Bots](/t/bot).

<op>list bot</op>

```

```

### metric

`list metric` lists all [Metrics](/t/metric).

<op>list metric</op>

```

```

### resource

`list resource` lists all [Resources](/t/resource).

<op>list resource</op>

```

```


List

List Command


### Linux commands

Op supports the execution of Linux commands on resources specified by an Op resource query command

Example:

<op>host | .pod | .container | `ps aux`</op>

### Metric query

This query retrieves metric data on all the hosts from a prior resource query.  If there is no prior resource query, Op will retrieve metric data on all hosts.

Examples:

<op>host | cpu_usage | real_time = true | window(10s)</op>
<op>container | metric_query(metric_names="container_cpu_usage_seconds_total") / 60 | real_time = true | window(10s)</op>

### Resource commands

The resources() and resource_path_entry() commands are used to define resource queries. In the example below, we show how we simplified resource queries (for example resources(type="HOST") | resource_path_entry(type="POD")) by using predefined  symbols.  The default symbol "host" was created like so: resource host = resources(type="HOST"), and .pod will be replaced with resource_path_entry(type="POD").

Examples:

<op>resources(type="HOST")</op>

<note>
The above command is typically written as `host` using substitution.
</note>

<op>resources(type="HOST") | resource_path_entry(type="POD") (typically written as host | .pod using substitution)</op>
<op>host | limit=2</op>
<op>pod | namespace="istio-system"</op>


Commands

Linux Commands, Metric Queries, and Resource Commands


When monitoring and trouble-shooting, we typically work with data that is focused around a specific period of time.  We look for indicators and patterns that give us clues as to what potentially led up to and caused an incident.  The Op pipe operator (similar to the commonly used pipe operator from the Bash shell) allows us to filter, aggregate, and transform the results of our resource and metric queries.

For example:

<op>host | metric_query (...) | sum(30)</op> returns values from a window size of 30 data points and calculates the sum.
<op>host | limit=2</op> gets resources of type HOST but limits the result to two resources
<op>cpu_usage | real_time=true | window(30s)</op> gets cpu usage from the last 30 seconds

When parameters are piped to metric and resource queries, they are viewed as functions; hence, we refer to them as transform functions.  Op has a very robust set of transform functions described in the following sections.

It’s important to note that when you apply transform functions to a metric query/defined metric interleaved with arithmetic/comparison, you must use parentheses to make it clear what the operands of the arithmetic expression are and what the transform functions are applying to.

For example, instead of:

- `metric_query(...) | sum(1) | window(2s) + metric_query(...) | sum(3) | window(4s)`

You would use parentheses to group the operands to the + operator and make it explicit what the transform functions are applying to and use this:

- `(metric_query(...) | sum(1) | window(2s)) + (metric_query(...) | sum(3) | window(4s))`

So, if you wanted to then apply a transform function, like mean(5) to the result of the entire arithmetic expression, you would wrap the entire arithmetic expression in parentheses and apply the mean(5) like this:

- `((metric_query(...) | sum(1) | window(2s)) + (metric_query(...) | sum(3) | window(4s))) | mean(5)`

If you wanted to break the expression into smaller bite size chunks with fewer parentheses, you could create defined metrics for incremental components of the computation and glue them together, i.e define the following in the below order:

- `metric m1 = metric_query(...) | sum(1) | window(2s)`
- `metric m2 = metric_query(...) | sum(3) | window(4s)`
- `metric m3 = m1 + m2`
- `metric m4 = m3 | mean(4)`

## Time Aggregates

The time aggregate transform functions take 3 parameters:

- `aggregation_window_size`
- `window_mode` _(optional)_
- `drop_incomplete` _(optional)_

Aggregation Window Size: Each of these functions takes an aggregation window size (number of data points) as an argument. For example, if timestamps are \[0, 1000, 2000, 3000, 4000, 5000\] and values are \[0, 1, 2, 3, 4, 5\] then the sum aggregate with aggregation window size 3 gives timestamps \[2000, 5000\] and values \[3, 12\] (\[0 + 1 + 2, 3 + 4 + 5\]). A user would invoke this function by writing `host | metric_query(...) | sum(3)` to aggregate with aggregation function sum and aggregation window size 3.

Window Mode: Either “SLIDING” or FIXED”. The default is “SLIDING” if not specified. Setting window_mode="FIXED" results in the aggregation function being computed over non-overlapping blocks of data, each with aggregation_window_size data points. For example, `sum(5, "FIXED")` computes the sum over data points 1-5, 6-10, 11-15...etc. If window_mode="SLIDING", the aggregation function is computed over overlapping blocks of data, each with aggregation_window_size_data_points. For example, `sum(5, "SLIDING")` computes the sum over data points 1-5, 2-6, 3-7...etc.

Drop Incomplete: Either true or false. The default is false if not specified. Consider what happens if there are 10 data points in a series, and a sum time aggregate is applied to this series with an aggregation window size of 3 and a "FIXED" window mode. The last aggregation bucket will only contain a single data point, because 10/3 yields a remainder of 1. If `drop_incomplete=true`, then this last partially filled aggregation bucket will be dropped, and the aggregated series will only contain 3 data points (the sums over the first three buckets). If `drop_incomplete=false`, the aggregated series will contain 4 data points, because the sum will be computed over the first 3 full buckets along with the last partially filled bucket.

Here is example syntax for a sum time aggregate with all 3 parameters specified:  
`sum(aggregation_window_size=5, window_mode="FIXED", drop_incomplete=true)`. You can specify the parameters without their names, but they must be in the following order: \[aggregation_window_size, window_mode, drop_incomplete\]. Example syntax with all 3 parameters specified without name: `sum(5, "FIXED", true)`.

| Time Aggregate Name and Syntax                                      | Aggregation Function                                                                                         | Example                |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------- |
| `sum(aggregation window size[, window mode][, drop incomplete])`    | computes the sum of values across the given aggregation window                                               | cpu_usage \| sum(5)    |
| `count(aggregation window size[, window mode][, drop incomplete])`  | computes the number of values across the given aggregation window                                            | cpu_usage \| count(5)  |
| `irate(aggregation window size[, window mode][, drop incomplete])`  | computes the per second rate of change between the values at the beginning and end of the aggregation window | cpu_usage \| irate(5)  |
| `mean(aggregation window size[, window mode][, drop incomplete])`   | computes the mean of values across the given aggregation window                                              | cpu_usage \| mean(5)   |
| `max(aggregation window size[, window mode][, drop incomplete])`    | computes the max of values across the given aggregation window                                               | cpu_usage \| max(5)    |
| `min(aggregation window size[, window mode][, drop incomplete])`    | computes the min of values across the given aggregation window                                               | cpu_usage \| min(5)    |
| `stddev(aggregation window size[, window mode][, drop incomplete])` | computes the standard deviation of values across the given aggregation window                                | cpu_usage \| stddev(5) |
| `p1(aggregation window size[, window mode][, drop incomplete])`     | computes the 1st percentile of values across the given aggregation window                                    | cpu_usage \| p1(5)     |
| `p10(aggregation window size[, window mode][, drop incomplete])`    | computes the 10th percentile of values across the given aggregation window                                   | cpu_usage \| p10(5)    |
| `p90(aggregation window size[, window mode][, drop incomplete])`    | computes the 90th percentile of values across the given aggregation window                                   | cpu_usage \| p90(5)    |
| `p99(aggregation window size[, window mode][, drop incomplete])`    | computes the 99th percentile of values across the given aggregation window                                   | cpu_usage \| p99(5)    |

## Transforms

These functions change the data returned by a metric query (ex: cpu_usage, mem_usage, disk_available, packet_loss) by applying a function to each value in the series.

| Metric Query and Transform      | Transform Function                                                                                                                      | Example                        |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| `<metric_query> \| floor`       | computes the floor of each value in the time series                                                                                     | cpu_usage \| floor             |
| `<metric_query> \| ceil`        | computes the ceiling of each value in the time series                                                                                   | cpu_usage \| ceil              |
| `<metric_query> \| upper_bound` | computes the maximum of its argument and each value in the time series                                                                  | cpu_usage \| upper_bound(90.0) |
| `<metric_query> \| lower_bound` | computes the minimum of its argument and each value in the time series                                                                  | cpu_usage \| lower_bound(15.7) |
| `<metric_query> \| limit`       | takes a number as an argument and limits all the time series in the result to have a number of points less than or equal to this number | cpu_usage \| limit(4)          |
| `<metric_query> \| shift`       | shifts the values in the time series by amount given by its argument                                                                    | cpu_usage \| shift(-2)         |

<note>

For the `shift` transform given timestamps `[0, 1000, 2000, 3000, 4000]` and values `[0, 1, 2, 3, 4]`, piping into `shift(2)` gives timestamps `[2000, 3000, 4000]` and values `[0, 1, 2]`, while piping into `shift(-2)` gives timestamps `[0, 1000, 2000]` and values `[2, 3, 4]`.
</note>

## Parameters

| Metric Query Parameter Name                       | Function                                                                                                                                                                                           | Example                                 | Description                                              |
| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | -------------------------------------------------------- |
| window(left end parameter[, right end parameter]) | Window function specifies the range of time (left end to right end) for the Op statement. If the optional right range parameter is not specified, the right end of the time range defaults to now. | `cpu_usage \| window(30s)`              | gets cpu_usage from 30 seconds ago to now                |
| from, to                                          | limits the metric query from time from to time to                                                                                                                                                  | `cpu_usage \| from=1000 \| to=3000`     | gets cpu_usage from time 1000 to time 3000               |
| base, offset                                      | "If offset is positive, limit the query to cover the interval (base, base + offset) and if offset is negative, limit the query to cover the interval (base + offset, base)"                        | `cpu_usage \| base=3000 \| offset=2000` | gets cpu_usage from time 3000 to time 3000 + 2000 = 5000 |
| resolution                                        | sets the resolution of the data queried for, allowed resolutions are 1 sec, 10 seconds, 1 min and 1 hour.                                                                                          | `cpu_usage \| resolution=60`            | gets data at one minute resolution                       |

## Resource Aggregates

These functions aggregate metric query results across resources. For example, to get the sum of `cpu_usage` across all hosts, you would write `host | cpu_usage | r_sum`.

| Metric Query Aggregate Name | Function                                                                                                                        | Example                 |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
| r_sum                       | aggregates the metric query across resources by taking the sum of the values across each resource for each timestamp            | `cpu_usage \| r_sum`    |
| r_count                     | aggregates the metric query across resources by counting the number of values across each resource for each timestamp           | `cpu_usage \| r_count`  |
| r_mean                      | aggregates the metric query across resources by taking the mean of the value for each resource for each timestamp               | `cpu_usage \| r_mean`   |
| r_max                       | aggregates the metric query across resources by taking the max of the value for each resource for each timestamp                | `cpu_usage \| r_max`    |
| r_min                       | aggregates the metric query across resources by taking the min of the value for each resource for each timestamp                | `cpu_usage \| r_min`    |
| r_stddev                    | aggregates the metric query across resources by taking the standard deviation of the value for each resource for each timestamp | `cpu_usage \| r_stddev` |
| r_p1                        | aggregates the metric query across resources by taking the 1st percentile of the value for each resource for each timestamp     | `cpu_usage \| r_p1`     |
| r_p10                       | aggregates the metric query across resources by taking the 10th percentile of the value for each resource for each timestamp    | `cpu_usage \| r_p10`    |
| r_p90                       | aggregates the metric query across resources by taking the 90th percentile of the value for each resource for each timestamp    | `cpu_usage \| r_p90`    |
| r_p99                       | aggregates the metric query across resources by taking the 99th percentile of the value for each resource for each timestamp    | `cpu_usage \| r_p99`    |

## Resource Query Parameters

These parameters limit the results of a resource query via pipes. For example, the piped query `host | limit=2` gets resources of type HOST and limits the result to two resources.

| Resource Query Parameter Name | Function                                                                                                                                                                                                                 | Example                                 | Description                                                                    |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------- | ------------------------------------------------------------------------------ |
| type                          | specifies the type of resources query is for                                                                                                                                                                             | `resources \| type="POD"`               | gets resources of type POD                                                     |
| from, to                      | limits the resource query from time from to time to                                                                                                                                                                      | `host \| from=1000 \| to=3000`          | gets the resources that existed between 1000 and 3000                          |
| base, offset                  | if the offset is positive, limits the query to cover the interval (base, base + offset) and if offset is negative, limit the query to cover the interval (base + offset, base)                                           |                                         |                                                                                |
| limit                         | limits the number of results of the resource or a combination of resources (referred to as tuples) query.                                                                                                                | `host \| .pod \| .container \| limit=5` | limits the result and returns at most 5 tuples of type (host, pod, container). |
| random                        | limits the number of results of the resource query by selecting random=0 or random=1. Random can be either 0/1. If random is 1, then the resources return will be shuffled (returned in random order.) 0 is the default. | `host \| random=1 \| limit=10`          | results in 10 random hosts.                                                    |
| name                          | only include resources with name the same as its argument                                                                                                                                                                | `host \| name="example"`                | gives all of the hosts with name "example"                                     |


Transformations

Transform Functions for the Metric Query


| Shortcuts | Description                                             |
| --------- | ------------------------------------------------------- |
| Ctrl + A  | Go to the beginning of the line (Home)                  |
| Ctrl + B  | Backward one character                                  |
| Ctrl + C  | Cancel current running query                            |
| Ctrl + D  | Delete character under the cursor or exits Op at prompt |
| Ctrl + E  | Go to the end of the line (End)                         |
| Ctrl + F  | Forward one character                                   |
| Ctrl + H  | Delete character before the cursor (Backspace)          |
| Ctrl + K  | Cut the line after the cursor to the clipboard          |
| Ctrl + L  | Clear the screen                                        |
| Ctrl + N  | Next command (Down arrow)                               |
| Ctrl + P  | Previous command (Up arrow)                             |
| Ctrl + R  | History search of executed commands (backward)          |
| Ctrl + S  | History search of executed commands (forward)           |
| Ctrl + U  | Cut the line before the cursor to the clipboard         |
| Ctrl + W  | Cut the word before the cursor to the clipboard         |


Keyboard Shortcuts

Op Keyboard Shortcuts


## count

The `count` function sums the number of [Resources](/t/resource) returned from the preceding query.

<op>host</op>

```
 ID | TYPE | NAME                | REGION    | AZ
 1  | HOST | i-052a365e984d32cc8 | us-west-2 | us-west-2a
 2  | HOST | i-0029b1ae1177ed57a | us-west-2 | us-west-2a
 3  | HOST | i-0790fc0b3b31278c5 | us-west-2 | us-west-2a
 4  | HOST | i-093bd00fe05f465a5 | us-west-2 | us-west-2a
 5  | HOST | i-099afa4a819cd99df | us-west-2 | us-west-2a
```

<op>host | count</op>

```
 RESOURCE_COUNT
 5
```

## disk_used

The `disk_used` function returns the percent of disk used, excluding `rootfs`, for host [Resources](/t/resource) returned from the preceding query.

<op>host | disk_used</op>

```
 ID | TYPE | NAME                | REGION    | AZ         | TIMESTAMPS          | DISK_USED
 1  | HOST | i-052a365e984d32cc8 | us-west-2 | us-west-2a | 2021/06/09 22:54:28 |     89.86
    |      |                     |           |            | 2021/06/09 22:54:29 |     89.86
    |      |                     |           |            | 2021/06/09 22:54:30 |     89.86
    |      |                     |           |            | 2021/06/09 22:54:31 |     89.86
    |      |                     |           |            | 2021/06/09 22:54:32 |     89.86
 2  | HOST | i-0029b1ae1177ed57a | us-west-2 | us-west-2a | 2021/06/09 22:54:28 |     90.63
    |      |                     |           |            | 2021/06/09 22:54:29 |     90.63
    |      |                     |           |            | 2021/06/09 22:54:30 |     90.63
    |      |                     |           |            | 2021/06/09 22:54:31 |     90.63
    |      |                     |           |            | 2021/06/09 22:54:32 |     90.63
```

## TODO

| Function | Description                                                      |
| -------- | ---------------------------------------------------------------- |
| auth     | `auth <url>` - authenticates users via Okta to access Shoreline. |

Valid auth url: `https://<customer>.<region>.api.shoreline-<cluster>.io`
Ex: https://cust1.us.api.shoreline-prod.io
`auth <url><token> `- allows manual import of an authorization token (e.g. when a browser is not available
`auth list` - list all stored auth credentia
`auth refresh` - update the current auth credential
`auth switch` - select different stored auth credential
`auth vars` - variables that can be used to set custom cli prompt or custom cli window title.
Ex: config set cli.prompt "op:${{auth.customer}}-${{auth-region}}-${{auth.cluster}}>"
Ex: config set cli.title "op:${{auth.user}}-${{auth.customer}}-${{auth-region}}-${{auth.cluster}}>" |
| columns | Shows available columns that you can show or hide in the cli for the resource that you are querying. There is a corresponding config setting for each available column for you to bring in to the cli
Ex: columns host \| .pod |
| config | config list - list all available config settings
config get - list a single config setting
Ex: config get timer
config set - change the config setting
Ex: config set table.sections.linux_cmd.format.stdout '{type: string, spec: "%s", width_max: 200, width_style: wrap }'
`config show <section.column_name>` - display the specified column
Ex: config show resources.fields.instance_type
config hide - hide the specified column
Ex: config hide resources.fields.az
`config shell "<path-to-shell>"` - set the shell for remote execution
Ex: config shell "/bin/sh" |
| count | Output the number of resource specified
Ex: host \| count
Ex: host \| .pod \| .container \| count |
| get_user_sessions | Get active user sessions in Shoreline. Output includes user, agent (cli or web), active time, and session id |
| help | List help for commands and symbols
`help <command>` - prints help for single command |
| `history` | List commands executed in order during the current session |
| `log` | log true - enables logging to "op-session.log" in the current directory. All commands and responses are written to the file, as well as the console
log false - disables logging
`log <file name>` - enables logging to a specific destination file |
| op | op replay - re-displays the output for the last Op command, applying any config changes. Does not re-execute the command
`op save <jsonfile>` - saves the returned JSON result for the last Op command to a file
`op load <jsonfile>` - loads a saved JSON result from a file to replay. |
| source | Run the op commands sequentially in a flat file, aborting on any errors
Ex: source maintenance_script |
| version | Print the version of the cli |

```

```


Functions

Op Functions


## Prerequisites

- Local Terraform executable installed
- OR, Terraform Cloud account

Authorization from:

`.ops_auth.yaml`

Eventually from Shoreline API endpoint.

## Preview

- `.terraformrc` not packaged, but because we are not on the public package repo
- `/tmp/tf-repo/providers`

- pulls `ops_auth` file and uses current api
- User specifies URL for auth
  - If unable to auth perhaps auto-open browser

## Resources

- actions
- arlarms
- bots
- metrics
- files (i.e. a copy in the CLI)

Properties

- `enabled`: Default is `false`

## Terraform

- a difference engine
- a template expansion engine

- `terraform plan`
- `terraform apply`
  - confirm
- `terraform destroy`


Terraform

Op Packs

Packs

Administration

Access Control

Audit


## Data Refresh

Shoreline periodically ingests all of the [Resource](/t/resource) data collected by [Agents](/t/agent). The [Monitoring UI](/t/ui/monitoring) displays the aggregated [Resource](/t/resource) data.

- `Real time`:
- `Recent`: Only available for


Monitoring

Monitoring Dashboard

Shoreline UI


<youtube id="RkN9aiEp7FI" />

## Dimensions

<op>host | .pod</op>

Use [Dimensions](https://shoreline.helpdocs.io/article/8oa6ylqzgl-configuring-dimensions) to adjust the resource specificity within the view. Click a **Dimension** to drill up or down with options such as **Region**, **AZ**, **Host Name**, **App**, and **Name Space**.

![dimensions](/assets/images/ui/metrics-explorer/dimensions.png)

- Click on a **Dimension** `+` sign to enable it.
- Alternatively, click a **Dimension** `-` sign to disable it.
- Enabling **Dimensions** loads charts automatically for all filtered resources.

### Aggregation

Anytime you are viewing combined **Dimensions** the displayed resource charts are automatically aggregated.

For example, here we're viewing both the **Region** and **AZ** **Dimensions**, but not the **Host Name**. We have (2) hosts within the `us-east-1b` AZ, so the `us-east-1b` chart shows the number of hosts and automatically aggregates data within that AZ:

![dimensions-aggregated](/assets/images/ui/metrics-explorer/dimensions-aggregated.png)

## Filters

**Filter** dropdowns allow you to adjust multiple facets of the displayed data.

![filters](/assets/images/ui/metrics-explorer/filters.png)

### Resources

The **Resources** filter displays all resource types within the environment. Select the **Resource** type to view related data.

- **Host**: Displays all hosts within the environment.
- **Pod**: Displays all pods.
- **Container**: Displays all containers.

![filter-resources](/assets/images/ui/metrics-explorer/filter-resources.png)

### Metrics

The **Metrics** dropdown lists all [Metric Sets](https://shoreline.helpdocs.io/article/q6c4jvfr25-configuring-metric-sets), which are based on the currently selected **Resource**.

![filter-metrics](/assets/images/ui/metrics-explorer/filter-metrics.png)

**Metric Explorer** includes many default [Metric Sets](https://shoreline.helpdocs.io/article/q6c4jvfr25-configuring-metric-sets).

> **TIP**: Custom [Metric Sets](https://shoreline.helpdocs.io/article/q6c4jvfr25-configuring-metric-sets) can be created on the **Configuration > Metric sets** page.

#### Host Metric Sets

The default [Metric Sets](https://shoreline.helpdocs.io/article/q6c4jvfr25-configuring-metric-sets) for host **Resources** are:

- Core
- CPU
- Disk
- Filesystem
- Memory
- Network
- Vmstat

#### Pod Metric Sets

The default [Metric Sets](https://shoreline.helpdocs.io/article/q6c4jvfr25-configuring-metric-sets) for pod **Resources** are:

- Core
- Filesystem
- Memory
- Network

#### Container Metric Sets

The default [Metric Sets](https://shoreline.helpdocs.io/article/q6c4jvfr25-configuring-metric-sets) for container **Resources** are:

- Core
- Filesystem
- Memory

### Period

You can adjust the time period you're viewing by altering the **Period** dropdown.

For example, selecting **last 24 hours** will display all monitoring data from the last day.

![filter-metrics](/assets/images/ui/metrics-explorer/filter-period.png)

### Overlays

![filter-metrics](/assets/images/ui/metrics-explorer/filter-overlays.png)

### Columns

![filter-metrics](/assets/images/ui/metrics-explorer/filter-columns.png)

### Chart Type

![filter-metrics](/assets/images/ui/metrics-explorer/filter-chart-type.png)

---

# Original Text

## Step-by-Step Instructions

### Monitor by Dimensions

Dimensions are how you drill up or drill down

### Monitor by Time Period

You can view alarms for a specific time period

1.  Click on the drop down arrow next to **Period**
2.  Select **Last 7 days**, **Last 24 hours**, **Last 60 minutes**, or **Last 10 minutes**
3.  To define your own time period, click on **Custom**
4.  Select the **Start** and **End** dates and times and click **Apply**

Shoreline supports _per second_ metrics and depending on the time period selected, there is an automatic rollup that is performed to provide you the correct data points. For example:

- **10 minutes** gives you 600 data points
- For **Last 60 minutes**, you get what is called a 10-second _resolution_. That means that (10) 1-second metrics are aggregated to create a single data point for a 10-second window. That calculates to 6 data points per minute and 360 data points for the hour.
- For **Last 24 hours**, the resolution is 1 minute
- For **Last 7 days**, the resolution is 1 hour

### Refreshing Data

You can refresh data automatically or manually

1.  Click on the drop down arrow next to **Refresh**
2.  To refresh automatically, select **Recent** or **Real time** (which is the most up to date)
3.  Select the refresh Frequency - every **1**, **5**, **10**, **30** seconds, or **1 minute**
4.  To refresh manually, select **Manual** under **Refresh every**
5.  Click on the **Refresh** button
6.  A timestamp of the last refresh is displayed next to the **Refresh** button

**![](https://files.helpdocs.io/6m3dgskxth/articles/5wzpeh50f8/1609356294034/monitoring-screen-shot-5.png)**

> There is an interplay between the refresh and the resolution that is important to note. For example, with Last week, the resolution is 1 hour. However, if you select Real time or Recent with a frequency of a 30 second interval, because the resolution is 1 hour, the data will be refreshed at the top of the hour. Ex: 12 pm, 1 pm, 2 pm and so forth. If something happened at 12:15 pm, you will not see it until 1 pm when the data is refreshed.

### Aggregate Data

You can choose how you want to aggregate a rollup of data

- The options are percentiles, minimum, average, or maximum.

**![](https://files.helpdocs.io/6m3dgskxth/articles/5wzpeh50f8/1611889374157/monitoring-aggregate-options-screen-shot.png)**

### Graph Overlays

On top of the default metric graphic, you can draw additional overlay lines

- The options are **Statistics** and **Bands**
- Click on the **Alarm** overlay to see when Alarms were raised and resolved
- Alarm details are listed in the Annotations window
- Click on **Clear All** to remove an overlay

**![](https://files.helpdocs.io/6m3dgskxth/articles/5wzpeh50f8/1611889205010/monitoring-band-overlay.png)**

### 3, 2, 1 Column Display

- Charts can be displayed in 3, 2, or 1 columns

**![](https://files.helpdocs.io/6m3dgskxth/articles/5wzpeh50f8/1611888535438/monitoring-columns.png)**

### Ad Hoc Metrics

1.  Click on the magnifying glass in the upper right hand corner
2.  Enter the metric name or partial name
3.  Click on the metric and it will be added with an **X** next to the name
4.  Click on the **X** to delete the metric

**![](https://files.helpdocs.io/6m3dgskxth/articles/5wzpeh50f8/1611888430102/monitoring-ad-hoc.png)**

### Zoom In

1.  Highlight the area you want to zoom in on
2.  The graph will zoom in
3.  Repeat this to zoom in further
4.  Click **Reset zoom** to go back to the original display

**![](https://files.helpdocs.io/6m3dgskxth/articles/5wzpeh50f8/1611888331946/monitoring-zoom-1.png)**

**![](https://files.helpdocs.io/6m3dgskxth/articles/5wzpeh50f8/1611888342005/monitoring-zoom-2.png)**


Metric Explorer


## Shoreline (TODO)

- **Action:** Shell commands and shell scripts that act on your resources.
- **Agent:** The software agent installed on every managed node allows Shoreline to monitor connected resources.
- **Alarm:** Checks, built on metrics and shell commands, that determine if the state of your resources is an issue.
- **Bot:** Bind alarms and actions together: when an alarm fires, a mitigating action is automatically run to fix the issue
- **Metric:** Time series data associated with your resources such as CPU utilization, latency, throughput, or error rate.
- **Resource:** An individual infrastructure object within your environment.
- **Op Pack:**

## UI

- **Dashboard** or **Dashboard UI:** The root Shoreline UI page within the [UI](/t/ui).
- **Monitoring UI:** The [Monitoring UI](/t/ui/monitoring) page within the [UI](/t/ui).
- **Alarms UI:**
- **Actions UI:**
- **Resources UI:**
- **Configuration UI** or **Configuration:**
-


Glossary


## Capitalization

Capitalization guidelines:

- Use sentence-style capitalization: Capitalize the first word of a sentence, heading, title, UI element, or standalone phrase.
- Capitalize [proper nouns](#proper-nouns).
- Don't use all uppercase for emphasis.
- Don't use all lowercase as a design choice.

### Proper Nouns

A proper noun might be the name of a person, a brand, a product, a tool, a UI element, and so forth.

### Examples

- Shoreline, GitHub: Brand names always follow their respective capitalization conventions, including internal capitalization.
- Amazon Web Services (AWS), Google Cloud Platform (GCP): Always capitalized acronyms. See [Acronyms](#acronyms) for more info.
- Agent, Alarm, Action, Bot, UI: References to Shoreline tools and services that use common words are always considered proper nouns in their intended context. Always capitalize references to Shoreline products and tools.

## Acronyms

In general, only use acronyms if they're more recognizable than what they stand for.

For example, FAQ and URL are acceptable acronyms.

For lengthy or frequently-acronymed terms, fully spell out the term followed by a parenthetical acronym the first time the article uses it. Then, future references can utilize the acronym only.

For example, write Amazon Web Services (AWS) on the first mention, then just AWS on future references.


Style Guide

Writing Guide

Scratchpad

Internal


Shoreline was founded by Anurag Gupta in February 2019. Anurag was the VP and GM for Analytics and Database Services at Amazon Web Services (AWS). In addition to Anurag, the Shoreline team has strong operational experience from AWS, Apple and Cisco, where they managed large scale cloud applications and infrastructure. Shoreline is headquartered in Redwood City, California.

## Customer Problems Addressed by Shoreline

A significant majority of service related incidents in the cloud are well understood and can be addressed with automated remediation. Unfortunately, operators at many businesses with applications and services in public clouds such as AWS, GCP or Azure, spend a majority of their time addressing incidents manually, causing unnecessary ticket-related fatigue in operations teams. In addition, the mean-time-to-repair for many of the incidents remains high, breaking any customer related SLAs and internal SLOs.

Shoreline’s automated remediation platform increases service availability and reduces load on the operations teams. Shoreline’s platform takes care of most common issues quickly and automatically. In addition, operators can use Shoreline to debug and find the root cause for those occasional “first-time” issues. Shoreline does this by using actions that are easy for operators to create, and that are executed automatically by triggers based on processing of high resolution metrics collected in real time from customers’ applications and infrastructure.

## Typical Shoreline Customer

A typical Shoreline customer is a company that is running a large share or all of its digital services and applications in one or more more public clouds like Amazon’s AWS, Microsoft’s Azure or Google’s GCP. The company may have compute instances (nodes) ranging from a few hundred to a few thousand in these public clouds, supporting thousands of services end users. Such a company would use Shoreline to remediate issues automatically and quickly by deploying Shoreline’s agents on their nodes and using the Shoreline suite of SaaS applications to manage their services.

## How the Shoreline Solution Works

**Efficient agents with built-in automation** - agents are installed by the customers on their compute nodes. These agents scrape metrics every second from infrastructure and application, compresses them and stores them locally. To ensure the metrics are stored in reliable storage, the agents also “push“ these compressed metrics out to Shoreline service running in the cloud. The agents can also execute actions (snippets of code) that can take corrective actions on the node.

**Monitoring for remediation** - monitoring is done through a web-based graphical user interface. It helps users visually understand their infrastructure/services and the underlying metrics collected from them. Users can also see and monitor alarms in a single place. Like metrics, any alarms that have already been fired can be filtered by resources, services and by different time window.

**Remediation Actions** - users can create specific actions using the Shoreline Op Language, a language purpose built to perform specific actions fleet-wide or on specific resources based on metrics and associated alarms they generate.

**Bots** - bots allow users to connect named alarms to named actions to create automated remediations - "bots." This feature enables Shoreline to detect and remediate issues quickly, without human intervention

## Architecture

**Hardware** - all system, security, and network infrastructure are hosted in Amazon Web Services (AWS).

**Software** - the Shoreline agent (collects metrics and performs actions) and Saas service (manages metrics and performs operational activities) are proprietary and open source.

## Security

**Physical** - the Shoreline system and supporting infrastructure is hosted on AWS. As such, AWS is responsible for the physical security controls for the system.

**Network** - firewall systems are in place to filter unauthorized inbound network traffic from the Internet and deny any type of network connection that is not explicitly authorized. Network address translation (NAT) functionality is utilized to manage internal IP addresses. Administrative access to the firewall is restricted to authorized employees.

**Access** - Shoreline uses role-based security architecture and requires users of the system to be identified and authenticated prior to the use of any system resources. Resources are protected through the use of native system security and add-on software products and services that identify and authenticate users and validate access requests against the users’ authorized roles in access control lists. Employees and approved vendor personnel sign on to Shoreline infrastructure and services using either SSO or VPN, combined with two factor authentication.

## Operations

**Backups** - all customer data is backed up every hour to Amazon’s S3 storage service. All data is encrypted at rest and in transit. If any backups fail, personnel are notified appropriately.

**Availability** - Shoreline monitors the capacity utilization of physical and computing infrastructure both internally and for customers to ensure that service delivery matches customer expectations. Shoreline evaluates the need for additional infrastructure capacity in response to growth of existing customers and/or the addition of new customers.


Op Functions Introduction


The List command returns all instances of the object type specified - resource, metric, alarm, action, bot, or all. The object can be further qualified by piping (adding) an attribute name.

Syntax: `list <object_name> | <filter>`

Below are the `list` commands which Op supports:

<op>list all</op>
<op>list actions</op>
<op>list alarms</op>
<op>list bots</op>
<op>list files</op>
<op>list metrics</op>
<op>list resources</op>


List Function


## TODO

| Function | Description                                                      |
| -------- | ---------------------------------------------------------------- |
| auth     | `auth <url>` - authenticates users via Okta to access Shoreline. |

Valid auth url: `https://<customer>.<region>.api.shoreline-<cluster>.io`
Ex: https://cust1.us.api.shoreline-prod.io
`auth <url><token> `- allows manual import of an authorization token (e.g. when a browser is not available
`auth list` - list all stored auth credentia
`auth refresh` - update the current auth credential
`auth switch` - select different stored auth credential
`auth vars` - variables that can be used to set custom cli prompt or custom cli window title.
Ex: config set cli.prompt "op:${{auth.customer}}-${{auth-region}}-${{auth.cluster}}>"
Ex: config set cli.title "op:${{auth.user}}-${{auth.customer}}-${{auth-region}}-${{auth.cluster}}>" |
| columns | Shows available columns that you can show or hide in the cli for the resource that you are querying. There is a corresponding config setting for each available column for you to bring in to the cli
Ex: columns host \| .pod |
| config | config list - list all available config settings
config get - list a single config setting
Ex: config get timer
config set - change the config setting
Ex: config set table.sections.linux_cmd.format.stdout '{type: string, spec: "%s", width_max: 200, width_style: wrap }'
`config show <section.column_name>` - display the specified column
Ex: config show resources.fields.instance_type
config hide - hide the specified column
Ex: config hide resources.fields.az
`config shell "<path-to-shell>"` - set the shell for remote execution
Ex: config shell "/bin/sh" |
| count | Output the number of resource specified
Ex: host \| count
Ex: host \| .pod \| .container \| count |
| get_user_sessions | Get active user sessions in Shoreline. Output includes user, agent (cli or web), active time, and session id |
| help | List help for commands and symbols
`help <command>` - prints help for single command |
| `history` | List commands executed in order during the current session |
| `log` | log true - enables logging to "op-session.log" in the current directory. All commands and responses are written to the file, as well as the console
log false - disables logging
`log <file name>` - enables logging to a specific destination file |
| op | op replay - re-displays the output for the last Op command, applying any config changes. Does not re-execute the command
`op save <jsonfile>` - saves the returned JSON result for the last Op command to a file
`op load <jsonfile>` - loads a saved JSON result from a file to replay. |
| source | Run the op commands sequentially in a flat file, aborting on any errors
Ex: source maintenance_script |
| version | Print the version of the cli |


Other Functions


## What is Op?

As operators, you perform many manual tasks to maintain, monitor, and troubleshoot applications in the production environment.  If you are lucky, these tasks are part of scheduled maintenance operations:  a predetermined time block is published, users are notified, systems are quiesced, you go in and do your thing and get out, bring the system back up and notify everyone.

However, more often than not you have to do these manual tasks on-demand to handle a page during an on-call rotation.  Your organization may pursue the goal of NoOps (the idea that a software environment can be completely automated so that there's no need for an operations team to manage it) but we all know that fixing root causes in software takes weeks to months.  And during that time, we keep incurring and executing manual tasks to mitigate the issue.

At the same time, the number of supported environments, Clouds, and software deployments continues to increase, only further putting pressure on your ability to meet SLAs.

Shoreline Op is a purpose-built operations oriented language designed to allow operators and admins to rapidly:

- understand, debug, and fix systems during an operational event
- automate the tasks performed during mitigation in order to reduce or even end future manual processing

## How Op Works

While you (the operator) are debugging, the Op interactive CLI allows you to gather and correlate information about resources, metrics, and system state to give you greater insight and inform the actions you need to take to mitigate the issue.  Afterwards, Op allows the operator to encode behaviors into Actions, Alarms, and Bots to automatically search for and fix the issues, without operator manual intervention going forward.

Using the Op language for “in the moment” debugging and permanent fixing significantly reduces the time to automate.  The statements used during operational work to identify and remediate an issue can be immediately deployed for permanent issue detection and fix.

## The Op Language

Op uses familiar Bash syntax which makes it easy to learn. Its core primitives allow the operator to focus on encoding how to detect and mitigate an event rather than solving distributed systems problems.  Operators tell Op what to look for and how to fix it and then Op handles the undifferentiated heavy lifting such as propagation, error handling, auditing, system lease,...etc.

In the Op Language Guide articles, we will go through how to use Op for both interactive debugging and permanent auto remediation.  We've also published a Glossary that has full documentation for each of the Op commands.


Introduction to the Op Language


Op has five core types of objects:

- **Resources** - The infrastructure objects in your environment.
- **Alarms** - Checks, built on metrics and shell commands, that determine if the state of your resources is in issue.
- **Bots** - Bind alarms and actions together: when an alarm fires a mitigating action is automatically run to fix the issue
- **Actions** - Shell commands and shell scripts that act on your resources.
- **Metrics** - Time series data data associated with your resources such as CPU utilization, latency, throughput, or error rate.

We will start our Op walkthrough with Resources; each of these objects will be explained in detail in separate sections.

## Introduction to Resources

Resources can be hosts, pods, containers, virtual machines, database instances etc. On different platforms, we can have different resources e.g. pods and containers for Kubernetes and virtual machines for AWS, GCP, or Azure. Op explicitly models resources and their relationships. Let's start off with a simple resource query from the command line:

<op>host</op>

```
 ID | TYPE | NAME                | REGION    | AZ
 1  | HOST | i-064246cabcbb73067 | us-east-1 | us-east-1b
 2  | HOST | i-0812b3fd801632ecc | us-east-1 | us-east-1b
 3  | HOST | i-0189e804e46578b34 | us-east-1 | us-east-1a
```

Above you can see each of the host resources that appear in your environment. In this example, we have 3 AWS EC2 instances, all in the us-west-2 region, different availability zones. Each of these hosts is running a Shoreline agent. The agent is a program which communicates to the Shoreline backend, sends up telemetry and other environmental data, and receives various commands to carry out. In the above output, you can see each resource's name, region, and zone. Every object in Shoreline has a globally unique id and a human readable name.

Objects also have many other fields and you can configure their output. For example, hosts have tags that encode information such as geography or workload. We can interactively adjust this output for the session.

In this example, we have resources in AWS that have tags. Tags are name-value pairs that are used to categorize and organize resources. Let's print out the az tag (the availability zone tag, currently called “topology.kubernetes.io/zone”) for each host.

The resources_tags() function tells us what kind of tags our resources have. For example, running resources_tags(type=”HOST”) shows that we have a region and a zone tag for our hosts (among other tags). We can use this function to find existing tags and to better filter the resources we’re looking for.

<note>Currently, the CLI only outputs the JSON response for this function. Formatting the output is planned for a future release of Op.</note>

So how can you find a particular tag and filter by it and display the column in the CLI?

<op>config hide resources.fields.sl_TestLabel</op>

To change it permanently you can add it to the config (`.ops.yaml`), e.g.:

```yaml
table:
  sections:
    resources:
      show:
      - fields.kubernetes.io.arch
      hide:
      - fields.sl_TestLabel
```

<note>Slash characters (`/`) are converted to dots (`.`). There is also a provision for shortening resource tags (e.g. `kubernetes.io` -> `kube`).</note>

## Resource Definitions

Let's look at the alarms, actions and bots associated with your resources. To do this, we use the list command.  
<op>list alarm</op>

![](/assets/images/articles/v1/op-language/chapter-3-screen-shot-2.png)

<op>list action</op>

![](/assets/images/articles/v1/op-language/chapter-3-screen-3.png)

<op>list bot</op>

![](/assets/images/articles/v1/op-language/chapter-3-screen-shot-4.png)

For each of the above commands, we see the default output columns: name of the definition, description, and the formula. For alarms, we also see the more complex structure including resources to run the alarms on, and the clear query to resolve the alarm. The names of each of these objects is the name that we can refer to the object by. These names are globally unique.  
Above we ran a resource query to actually query for our resources. We can also get the definitions of our resource and metric queries, just like we do for alarms, actions, and bots:

<op>list resource</op>

```
 TYPE     | NAME      | FORMULA
 RESOURCE | host      | resources(type="HOST")
 RESOURCE | pod       | resources(type="POD")
 RESOURCE | container | resources(type="CONTAINER")
```

<op>list metric</op>

| TYPE   | NAME                        | DESCRIPTION    | FORMULA                                                               |
| ------ | --------------------------- | -------------- | --------------------------------------------------------------------- |
| METRIC | cpu_usage                   | Host cpu usage | (100 - 100 \* (metric_query(metric_names="node_cpu_seconds_total",... |
| METRIC | cpu_usage_pre               | test           | metric_query(metric_names="cpu_usage_pre")                            |
| METRIC | disk_available              |                | metric_query(metric_names="node_filesystem_avail_bytes")              |
| METRIC | network_throughput          |                | metric_query(metric_names="node_network_transmit_packets_total")      |
| METRIC | packet_loss                 |                | metric_query(metric_names="node_network_transmit_drop_total")         |
| METRIC | os_procs_running            |                | metric_query(metric_names="node_procs_running")                       |
| METRIC | os_procs_blocked            |                | metric_query(metric_names="node_procs_blocked")                       |
| METRIC | network_transmit_packets_v2 |                | metric_query(metric_names="node_network_transmit_packets_total")      |
| METRIC | network_transmit_bytes_v2   |                | metric_query(metric_names="node_network_transmit_bytes_total")        |
| METRIC | network_receive_bytes_v2    |                | metric_query(metric_names="node_network_receive_bytes_total")         |

This command lists the metric, its description, and underlying formula. As you can see in this example, Op allows you to define custom and complex metrics, specific to your environment.

Metric and resource definitions both support symbolic manipulation. As you can see above, the definition of host is actually resources(type="HOST"). What this means is that when you typed in host above, the host symbol was translated to that resource query. We will do much more of this later. Op fully supports user defined substitutions.

## The Op Pipe Operator

The next major language feature of Op is the pipe '|' operator. The pipe operator looks just the commonly used operator from the Bash shell and it behaves in a similar way. For example, let's filter our hosts down to only those in the availability zone us-west-2b.  
Like the conventional pipe operator from Bash, the pipe operator allows us to define processing pipelines that filter our results as they are passed through. But this pipe operator has some other features as well. Let's combine together two different type expressions. Let's get all our hosts in us-west-2b and then get their current CPU utilization for the last 5 seconds:

<op>host | az = "us-west-2b" | cpu_usage | window(5s) | real_time=true</op>

```
 ID | TYPE | NAME                | REGION    | AZ         | TIMESTAMPS          | CPU_USAGE
 1  | HOST | i-064246cabcbb73067 | us-east-1 | us-east-1b | 2021/06/01 12:56:18 |      8.00
    |      |                     |           |            | 2021/06/01 12:56:19 |      8.50
    |      |                     |           |            | 2021/06/01 12:56:20 |      7.50
    |      |                     |           |            | 2021/06/01 12:56:21 |      8.50
 2  | HOST | i-0812b3fd801632ecc | us-east-1 | us-east-1b | 2021/06/01 12:56:18 |      5.50
    |      |                     |           |            | 2021/06/01 12:56:19 |      4.50
    |      |                     |           |            | 2021/06/01 12:56:20 |      6.75
    |      |                     |           |            | 2021/06/01 12:56:21 |      4.75
```

Op allows us to bring together different types of objects into a pipeline. Specifically, Op pipelines support everything an operator needs in order to do interactive debugging: resources, metrics, and shell commands. For example, to get all of the pods, the containers scheduled to the host, and what they are running for each host running over 80% cpu utilization, we would use the following command with 'ps':

<op>host | cpu_usage | window(1s) > 0.8 | .pod | .container | `ps aux`</op>

<vc version="next">

How did Op do this? We would use the explain command to see what Op did:

<op>explain host | cpu_usage | window(1s) > 0.8 | .pod | .container | `ps aux`</op>

Like SQL, _explain_ shows what Op would do, but doesn't actually run the statement. Instead, it shows which steps the plan would run, on which hosts, and it what order. A step is a single piece of work that is performed. Steps are dependent on each other; hence, they enforce an order of execution. Op handles the process of determining the appropriate plan for your statement by decomposing it into steps to carry out what you define. Each step has a sequence number. Each step depends on the execution of certain sequence numbers. As the steps execute, an Op object called a Scope is built up i.e. the scope is a summary of all the computation performed by the statement.

Here are the steps for the above plan:

- The backend service runs the resource query step to determine which hosts are in us-west-2a.
- A metric query step is run on each of these hosts to fetch the live CPU usage.
- For each host where the metric query exceeds 80%, a local resource query step is run to obtain the pods and containers. Finally, a Linux command step "ps aux" is run in each of these containers.

</vc>


Getting Started With Op


## Discovering Resources

One of the foundational things we need to understand as operators is how a workload is mapped to underlying infrastructure.  Getting that information is easy with one of the most simple and useful commands in Op:  Resource.  Resource allows us to query our environment to understand which hosts, pods, containers, and other resources are present.  For example, to get all of the hosts in our environment:

<op>host</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/kifc1qbtr1/1620094277033/screen-shot-2021-05-03-at-7-11-13-pm.png)

Similarly, you can retrieve all pods and containers in our environment:

<op>pod</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/kifc1qbtr1/1620094303982/screen-shot-2021-05-03-at-7-11-42-pm.png)

<op>container</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/kifc1qbtr1/1620094356703/screen-shot-2021-05-03-at-7-12-32-pm.png)

To retrieve all resources, you can run:

<op>resources</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/kifc1qbtr1/1620094723103/screen-shot-2021-05-03-at-7-18-38-pm.png)

Op resources are also aware of their relationships to other resources which enables you to traverse resource hierarchies by piping multiple resource queries into one statement. To traverse the resource hierachy, you need to prefix the resource with a dot (.)

To retrieve the pods that are scheduled to each host:

<op>host | .pod</op>

To retrieve all containers in each pod scheduled to each host:

<op>host | .pod | .container</op>

Op is all about not repeating ourselves.  “Fat fingering” incorrect commands is widely known to be the cause of some of the most disruptive outages.  Operators also don't have time in the heat of a severity 1 incident to remember long and complex commands.  When we have a useful command like the above in Op, we can easily create a shorthand for it by creating a named resource query.  In this example, we create a named query called “app_containers”.

<op>resource app_containers = host | .pod | .container</op>

Now we can run `host | .pod | .container` simply as:

<op>app_containers</op>

This shows the real power of Op.  The operator is able to build up, layer by layer, more and more powerful commands.

System reserved resource names or symbols such as host, pod, container, and resources cannot be modified by users

## Filtering and Drilling Down on Resources

Let's build on the `app_containers` query. We can filter the returned resources in Op not only by type, but also by tag.  For example, to get all of our containers in AZ us-west-2a we would run the following:

<op>app_containers | az = "us-west-2a"</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/kifc1qbtr1/1620097028840/screen-shot-2021-05-03-at-7-57-05-pm.png)

If we want to get all containers that are not in AZ us-west-2a, we would run:

<op>app_containers | az != "us-west-2a"</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/kifc1qbtr1/1620097349972/screen-shot-2021-05-03-at-8-02-24-pm.png)

If we wish to get all containers in AZ us-west-2a and us-west-2b, we would run the following:

<op>app_containers | az = ["us-west-2a", "us-west-2b"]</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/kifc1qbtr1/1620096946761/screen-shot-2021-05-03-at-7-55-43-pm.png)

Since op resources are also aware of their relationships to other resources, we can grab all of the containers in all of the pods on our hosts in us-west-2a and filter them to just Shoreline containers:

<op>host | az = "us-west-2a" | .pod | .container | container_name = "shoreline"</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/kifc1qbtr1/1620097463753/screen-shot-2021-05-03-at-8-04-20-pm.png)

## Advanced Resource Queries

### Set Operators

Using set operators (minus, union, intersect) you can define complex resource queries as a set.

To retrieve hosts in all AZs except us-west-2a:

<op>host minus (host | az = "us-west-2a")</op>

Same output as <op>host | az != "us-west-2a")</op>

To retrieve pods in us-west-2b and us-west-2c:

<op>(pod | az = "us-west-2b") union (pod | az = "us-west-2c")</op>

Same output as <op>(pod | az = ["us-west-2b","us-west-2c"]</op>

To retrieve Shoreline containers in us-west-2b:

<op>(container | az = "us-west-2b") intersect (container | container_name = "shoreline")</op>

Same output as <op>container | az = "us-west-2b" | container_name = "shoreline"</op>

### Resource Query with Regular Expressions (RegEx)

Op supports RegEx searches which use pattern matching to search for particular string of characters rather than constructing multiple, literal searches.

For example, if I want to get all containers which have "kube" in the name:

<op>container | name =~ "kube"</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/kifc1qbtr1/1620100537661/screen-shot-2021-05-03-at-8-55-32-pm.png)

Conversely, if I want to get all containers which does not have "kube" in the name:

<op>container | name !=~ "kube"</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/kifc1qbtr1/1620100650371/screen-shot-2021-05-03-at-8-57-24-pm.png)


Resources: Discovering the Environment


## Supported operating systems

- x86, 64 bit
- Linux: Centos, Ubuntu
- macOS
- Windows: not supported at this time

## Accessing Op

- Option 1 - cURL the latest release, make the executable, and copy to your path (or add to another directory and add that to your path)
- Option 2 - download from GitHub, make the executable, and copy to your path (or add to another directory and add that to your path)
- Option 3 - download from the Shoreline website, make the executable, and copy to your path (or add to another directory and add that to your path)

## Configuration

User specific configuration is stored in .ops.yaml in the user home directory.  The configuration file allows users to specify the following settings:

- Debug mode
- Output raw JSON
- CLI output table default format
- Additional formatting options (i.e. strings, numbers, time, labels)
- Aliases
- Linux commands

More details can be found here (external link needs to be added).

## Access

Access to the Shoreline Op server is initiated via the AUTH command in the CLI:

`op> auth <server url>`

The auth command takes you to your organization’s identity management service where you enter your credentials to login into the Shoreline Op server.

![](/assets/images/articles/v1/op-language/access-diagram.png)

## How to Execute Commands

To start using Op, simply run the Op command.  This starts the Op interactive CLI shell which is the primary interface for operators.

`op>`

You can also execute Op commands from text files and stdin.  They are standard shell commands and can be natively invoked from existing Bash scripts and shells.

You can set it to verbose mode for detailed explanations by using the -v flag.  We recommend that you use verbose mode when starting out with Op to accelerate your ramp-up and also to assist with debugging.  Verbose output is written out to stdout.


Op Tool Setup


In the **[Resources: Discovering the Environment](/v1/op-language/resources-discovering-the-environment)** article, we learned how to discover resources, filter on them, drill down for more detail, and learn what the relationships are.  But beyond just discovering the resources, we also want to know how our resources are operating.  We want to know host cpu, disk, memory, and network utilization.  We also want to know service latency and error rates.  To get this information, we need metrics. Op metrics are time series data with name, resource, and other tag metadata. Let's get the value for a metric right now.  Let’s get the average cpu utilization for each of our hosts for the last 30 seconds:

<op>host | cpu_usage | window(30s) | resolution=10 | real_time=true | mean(3)</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/0h85lt9lh6/1620105475097/screen-shot-2021-05-03-at-10-17-49-pm.png)

Op resources and metrics naturally mesh together.  Prefixing a metric query with a resource query narrows the metrics returned down to only those metrics associated with the returned resources.

## Creating New Metrics

Like resources, we can also define and save useful metric queries for later.  Let's define the concept of average cpu over an interval:

<op>metric cpu_2_min = cpu_usage | window(120s) | real_time =true</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/0h85lt9lh6/1620105655052/screen-shot-2021-05-03-at-10-20-40-pm.png)

As shown above, our definition statements are fully parameterized.  Op's macro system allows for expression of complex substitution, but with a familiar syntax.

Op also lets users create custom derived metrics. Let's create a new metric called "cpu_usage_new":

<op>metric cpu_usage_new = (100 - 100 \* (metric_query(metric_names="node_cpu_seconds_total", tags={'{'}"mode":"idle"{'}'}) | irate(2) | r_mean)) | lower_bound(0) | upper_bound(100)</op>

If you need to update the formula, simply over-write with the following:

<op>cpu_usage_new.formula = [new formula]</op>

To set unit of measurement (uom) for the metric:

<op>cpu_usage_new.unit = "percent"</op>

Now we can leverage both of the Op commands we have built up.  Let's get the average cpu utilization over the last 30 seconds for each of our hosts:

<op>host | cpu_2_min</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/0h85lt9lh6/1620105793098/screen-shot-2021-05-03-at-10-22-57-pm.png)

If the metric is always measure against a resource type, then you can associate a default resource type with the metric as follows:

<op>cpu_2_min.resource_type = "HOST"</op>

The above examples really show the power of Op.  Multiple layers of substitution allow you to very succinctly express a complex query.  And in the heat of a Sev 1 or similarly critical incident, you will be able to easily remember the commands and execute efficiently.

## List Existing Metrics

Op allows the user to list all previously defined metrics. Imagine you are interactively debugging your cluster using Op, and you want the cpu_2_min metric, defined above, on each of your hosts, but you don’t remember the name of the defined metric. Let’s list our existing metrics.

<op>list metrics</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/0h85lt9lh6/1620105979752/screen-shot-2021-05-03-at-10-26-14-pm.png)

To view just a metric by name:

<op>list metrics | name="cpu_usage_new"</op>

We see that we have an entry in the table for our cpu_2_min metric. We know what this metric represents since we defined it and provided it with an appropriate name. However, what if someone new to the ops team wants more information about the metric? Let’s add a description to our metric.

<op>cpu_2_min.description = "real time cpu usage over last 2 min"</op>

Now, if we list the metrics again:

<op>list metrics</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/0h85lt9lh6/1620106197944/screen-shot-2021-05-03-at-10-29-51-pm.png)

They will now see our metric description alongside the metric name and formula. Op supports standard CRUD (create, read, update, and delete) operations over metric definitions. Please refer to the Op Functions Glossary for more information regarding syntax and supported operations. These features enable an operations team to rapidly build up a shared statement bank of commonly used metrics, which allows for the dissemination of operations knowledge and the ability to quickly gather valuable system information without wasting time reinventing or misremembering commonly used metric formulas.

## Metrics Exporter Support

Op plugs directly into the **[Prometheus](https://prometheus.io/docs/introduction/overview/)** exporter ecosystem. Op can pull metrics from any Prometheus exporter as well as Prometheus itself.  Exporters such as Envoy and cAdvisor are auto-discovered and ingestible by the Shoreline agent.

For further information on Envoy, please see **[Envoy Overview](https://www.envoyproxy.io)**.

For examples of cAdvisor metrics, please see **[Monitoring container metrics using cAdvisor](https://prometheus.io/docs/guides/cadvisor)**.


Metrics: Understanding System Behavior

Dynamic Filter

Dynamic Filter: Endless Possibilities to Filter Relevant Resources

Bots: Preventing Toil


Querying metrics is useful, but as an operator, we often want to continuously monitor metrics, resources, and other system state.  We watch for dangerous metrics levels constantly by checking dashboards.  Typically, we run Kubectl, AWS, and Terraform commands to see which resources are deployed and we run shell commands to assess system health, on a manual basis.

Instead of manually performing these checks, we can create an alarm in Op.  Alarms can be defined on metrics, resources, and system state.  Let's create an alarm on a cpu metric that triggers whenever cpu usage exceeds 65% for over 15 seconds in a 30 second interval. The base Op command to configure this alarm looks like this and includes the following parameters:

<op>alarm high_cpu_alarm = (cpu_usage > 65 | sum(30)) >= 15.0</op>

- `Alarm Name` - must be alphanumeric, use underscores and/or dashes, and globally unique
- `fire_query` - this is the condition that triggers the alarm. It must be a valid Op statement.

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613940680893/screen-shot-2.png)

The above loads an alarm called high_cpu_alarm. But more details are needed (which are added by executing additional Op commands) to fully define and enable the alarm:

- `clear_query` - what condition clears the alarm?

<op>{'high_cpu_alarm.clear_query = (cpu_usage < 65 | sum(30)) >= 15.0'}</op>

- `resource_query` - what resources is this alarm acting upon (host, pod, or container)?

<op>high_cpu_alarm.resource_query = host resource_type</op>

- `resource_type` - what is the specific resource type (HOST, POD, CONTAINER)?

<op>high_cpu_alarm.resource_type = "HOST"</op>

- `raise_for` - only local alarm is currently supported

<op>high_cpu_alarm.raise_for = "local"</op>

- `raise_family` - to make this alarm interchangeble and actionable between the CLI and UI, add family type of "custom"

<op>high_cpu_alarm.raise_family = "custom"</op>

- `metric_name` - what specific resource metric is being monitored?

<op>high_cpu_alarm.metric_name = "cpu_usage"</op>

- `condition_type` - is the condition defined as above or below the threshold?

<op>high_cpu_alarm.condition_type = "above"</op>

- `condition_value` - what is the condition value?

<op>high_cpu_alarm.condition_value = "65"</op>

Executing these Op commands updates the alarm.

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613944985773/screen-shot-4.png)

By default, all defined objects in Op start in a disabled state.  We want to make sure that we only perform authorized and fully prepared computation.  Op is all about safety and reliability.  With the definition, clear condition, and resources statements, our alarm is ready to be deployed.  To do this, we enable our alarm so that it runs on each of our hosts, constantly checking if cpu usage has exceeded the threshold:

- `enable` - the default is false

<op>enable high_cpu_alarm</op>

To verify that the alarm is defined and enabled, we use the `list` command:

<op>list alarms | name="high_cpu_alarm"</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613946127232/screen-shot-5.png)

Our new alarm is also visible in the Shoreline UI. But it needs more information in order to be synchronized with and editable in the UI. This step is optional but highly so that operators can leverage both the CLI and UI seamlessly.

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613946825229/screen-shot-7.png)

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613946877063/screen-shot-6.png)

The Op commands to synchronize this alarm to the UI are:

- `description`

<op>high_cpu_alarm.description = "cpu_usage exceeds 65% 15s out of 30s"</op>

- `fire_title_template`

<op>high_cpu_alarm.fire_title_template = "cpu usage above threshold"</op>

- `fire_short_template`

<op>high_cpu_alarm.fire_short_template = "cpu usage above threshold"</op>

- `resolve_title_template`

<op>high_cpu_alarm.resolve_title_template = "cpu usage below threshold"</op>

- `resolve_short_template`

<op>high_cpu_alarm.resolve_short_template = "cpu usage below threshold"</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613947868572/screen-shot-8.png)

The alarm is now completely configured to be managed from both the CLI and the UI.

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613948212288/screen-shot-9.png)

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613948223026/screen-shot-10.png)

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613948246205/screen-shot-11.png)

To see all the configured alarms, use the `list` command.

<op>list alarms</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/rgiasn9uj0/1613949109227/screen-12.png)


Alarms: Determining When Something Goes Wrong


Thus far, we have learned how Op discovers resources and relationships, how to measure those resources are operating, and how to set up alarm when an event is triggered.  Monitoring and alarming are crucial activities, but they only tell us what or when something is going wrong.  The problems don't fix themselves.  However, as operators, we are responsible for mitigating and remediating these problems.

Instead of manually performing actions, Op lets us automate actions and encode an entire workflow. Let's create an action that displays the top 5 most cpu intensive processes and output the PID, command, cpu and memory usage.

The base Op command to configure this action includes the following parameters:

- `Action Name` - must be alphanumeric, use underscores and/or dashes, and globally unique
- `Command` - this is the shell command.

<op>action high_cpu_action =`ps aux`</op>

- `resource_query` - this must be a valid Op statement that takes action on Op resources

<op>high_cpu_action.resource_query = host | .pod | .container | app="shoreline" | limit=5</op>

To verify that the action is defined we use the _List_ command.

![](https://files.helpdocs.io/6m3dgskxth/articles/67uc28d8fp/1613953274296/screen-shot-13.png)

By default, all defined objects in Op start in a disabled state.  We want to make sure that we only perform authorized and fully prepared actions.  Op is all about safety and reliability.  To enable our action, use the \*enable\* command.

- `enable` - the default is false

<op>enable high_cpu_action</op>

Our new action is also visible in the Shoreline UI. But it needs more information in order to be synchronized with and editable in the UI. This step is optional but highly so that operators can leverage both the CLI and UI seamlessly.

![](https://files.helpdocs.io/6m3dgskxth/articles/67uc28d8fp/1613953720735/screen-shot-14.png)

![](https://files.helpdocs.io/6m3dgskxth/articles/67uc28d8fp/1613957670514/screen-shot-18.png)

The Op commands to synchronize this action to the UI are:

- `start_title_template`

<op>high_cpu_action.start_title_template = "high cpu remediation has started"</op>

- `start_short_template`

<op>high_cpu_action.start_short_template = "high cpu remediation has started"</op>

- `error_title_template`

<op>high_cpu_action.error_title_template = "high cpu remediation resulted in error"</op>

- `error_short_template`

<op>high_cpu_action.error_short_template = "high cpu remediation resulted in error"</op>

- `complete_title_template`

<op>high_cpu_action.complete_title_template = "high cpu remediation has completed"</op>

- `complete_short_template`

<op>high_cpu_action.complete_short_template = "high cpu remediation has completed"</op>

The action is now completely configured to be managed from both the CLI and the UI.

![](https://files.helpdocs.io/6m3dgskxth/articles/67uc28d8fp/1613957427212/screen-shot-16.png)

To see all the configured actions, use the _List_ command.

<op>list actions</op>

![](https://files.helpdocs.io/6m3dgskxth/articles/67uc28d8fp/1613957485129/screen-shot-17.png)


Actions: Fixing the Problem

Op Self-Tutorial

Op Language


## Automated Remediation

In the world of DevOps, operators respond to events and updates as they occur with the goal of making sure that systems are highly available and performant. You perform many manual tasks to monitor, and troubleshoot applications in the production environment. The number of supported environments, Clouds, and software deployments continues to increase, putting pressure on your ability to meet SLAs and maintain high availability.

Meeting your SLAs and maintaining high availability is a function of mean time to failure (MTTF) and mean time to repair (MTTR). MTTR for the majority of issues is at least 5x more than what is required to support SLAs. Detection, notification, and ticket routing takes time. But fixing the problem, not just observing and reporting on it, is what matters. And that’s where human intervention adds latency and further delays the resolution. Relying on manual intervention for change management and event handling is error-prone and cannot scale. You need automated monitoring and remediation to meet your organization’s SLAs.

## UI Managed Objects

**Resources** - resources are the infrastructure objects in your environment. Resources can be hosts, pods, containers, virtual machines, database instances etc. On different platforms, you can have different resources e.g. pods and containers for Kubernetes and virtual machines for AWS, GCP, or Azure.

**Metrics** - metrics are time series data associated with your resources such as CPU utilization, latency, throughput, or error rate. You can query the metrics gathered to monitor the environment. You also use metrics to determine if a condition or threshold has been met which requires an Action to be taken.

**Alarms** - alarms tell you when something has gone wrong and/or needs attention. Alarms are defined on metrics, resources, and system state and are raised when a condition is met.

**Actions** - actions are shell commands and shell scripts that help you mitigate and remediate the problem that an Alarm has identified. You can define custom actions and encode an entire operational workflow into a runbook using Actions.

**Bots** - bots bind Alarms and Actions together using IF-THEN-ELSE constructs. Bots specify the action or set of actions to take when the alarm fires.

## UI Single Pane of Glass

Shoreline is an end-to-end operations tool designed to automate the entire system monitoring and remediation lifecycle across:

- Kubernetes and EC2 Instances
- Zones and Regions
- AWS, Azure, and GCP

From a single pane of glass you can set up and continuously manage your entire environment.

![](/assets/images/articles/v1/ui/ui-lifecycle.png)

### Configuration

- Set up:
  - Metrics and resources
  - Monitoring and remediation actions
  - Scripts, and bots
  - Integrations to notification and ticketing services

### Monitoring

- Real time, drill-down, and multi-dimensional monitoring dashboard by:
  - Resources
  - Metrics
  - Tme period
- 1 second data sampling for alarm accuracy and to reduce false positives
- Ability to analyze from a granular host level up to a cluster

### Alarms

- Real time detailed notification of event by:
  - Alarm raised
  - Action execution
  - Alarm resolved
- Alarms trigger in 10 seconds
- Alarms can be set on combinations of metrics
- Alarms can be configured to be muted and/or confirmed based on real-time information to avoid false positives

### Remediation

- Notification and status of response to event
  - Action(s) taken
  - Bot executed
- Goal is to resolve issues within time period of seconds
- Actions and bots run locally and automatically

### Accessing the Shoreline UI

Access to the Shoreline UI is through a URL specific to your organization. You enter your credentials through your or Shoreline's identity management service to log into the Shoreline server.

![](/assets/images/articles/v1/ui/ui-access-diagram.png)


Introduction to the Shoreline UI


<youtube id="RkN9aiEp7FI" />

## Dimensions

<op>host | .pod</op>

Use [Dimensions](https://shoreline.helpdocs.io/article/8oa6ylqzgl-configuring-dimensions) to adjust the resource specificity within the view. Simply click a **Dimension** to drill up or down with options such as **Region**, **AZ**, **Host Name**, **App**, and **Name Space**.

![dimensions](/assets/images/ui/metrics-explorer/dimensions.png)

- Click on a **Dimension** `+` sign to enable it.
- Alternatively, click a **Dimension** `-` sign to disable it.
- Enabling **Dimensions** automatically loads charts for all filtered resources.

### Aggregation

Anytime you are viewing combined **Dimensions** the displayed resource charts are automatically aggregated.

For example, here we're viewing both the **Region** and **AZ** **Dimensions**, but not the **Host Name**. We have (2) hosts within the `us-east-1b` AZ, so the `us-east-1b` chart shows the number of hosts and automatically aggregates data within that AZ:

![dimensions-aggregated](/assets/images/ui/metrics-explorer/dimensions-aggregated.png)

## Filters

**Filter** dropdowns allow you to adjust multiple facets of the displayed data.

![filters](/assets/images/ui/metrics-explorer/filters.png)

### Resources

The **Resources** filter displays all resource types within the environment. Select the **Resource** type to view related data.

- **Host**: Displays all hosts within the environment.
- **Pod**: Displays all pods.
- **Container**: Displays all containers.

![filter-resources](/assets/images/ui/metrics-explorer/filter-resources.png)

### Metrics

The **Metrics** dropdown lists all [Metric Sets](https://shoreline.helpdocs.io/article/q6c4jvfr25-configuring-metric-sets), which are based on the currently selected **Resource**.

![filter-metrics](/assets/images/ui/metrics-explorer/filter-metrics.png)

**MetricExplorer** includes many default [Metric Sets](https://shoreline.helpdocs.io/article/q6c4jvfr25-configuring-metric-sets).

> **TIP**: Custom [Metric Sets](https://shoreline.helpdocs.io/article/q6c4jvfr25-configuring-metric-sets) can be created on the **Configuration > Metric sets** page.

#### Host Metric Sets

The default [Metric Sets](https://shoreline.helpdocs.io/article/q6c4jvfr25-configuring-metric-sets) for host **Resources** are:

- Core
- CPU
- Disk
- Filesystem
- Memory
- Network
- Vmstat

#### Pod Metric Sets

The default [Metric Sets](https://shoreline.helpdocs.io/article/q6c4jvfr25-configuring-metric-sets) for pod **Resources** are:

- Core
- Filesystem
- Memory
- Network

#### Container Metric Sets

The default [Metric Sets](https://shoreline.helpdocs.io/article/q6c4jvfr25-configuring-metric-sets) for container **Resources** are:

- Core
- Filesystem
- Memory

### Period

You can adjust the time period you're viewing by altering the **Period** dropdown.

For example, selecting **last 24 hours** will display all monitoring data from the last day.

![filter-metrics](/assets/images/ui/metrics-explorer/filter-period.png)

### Overlays

![filter-metrics](/assets/images/ui/metrics-explorer/filter-overlays.png)

### Columns

![filter-metrics](/assets/images/ui/metrics-explorer/filter-columns.png)

### Chart Type

![filter-metrics](/assets/images/ui/metrics-explorer/filter-chart-type.png)

---

# Original Text

## Step-by-Step Instructions

### Monitor by Dimensions

Dimensions are how you drill up or drill down

### Monitor by Time Period

You can view alarms for a specific time period

1.  Click on the drop down arrow next to **Period**
2.  Select **Last 7 days**, **Last 24 hours**, **Last 60 minutes**, or **Last 10 minutes**
3.  To define your own time period, click on **Custom**
4.  Select the **Start** and **End** dates and times and click **Apply**

Shoreline supports _per second_ metrics and depending on the time period selected, there is an automatic rollup that is performed to provide you the correct data points. For example:

- **10 minutes** gives you 600 data points
- For **Last 60 minutes**, you get what is called a 10-second _resolution_. That means that (10) 1-second metrics are aggregated to create a single data point for a 10-second window. That calculates to 6 data points per minute and 360 data points for the hour.
- For **Last 24 hours**, the resolution is 1 minute
- For **Last 7 days**, the resolution is 1 hour

### Refreshing Data

You can refresh data automatically or manually

1.  Click on the drop down arrow next to **Refresh**
2.  To refresh automatically, select **Recent** or **Real time** (which is the most up to date)
3.  Select the refresh Frequency - every **1**, **5**, **10**, **30** seconds, or **1 minute**
4.  To refresh manually, select **Manual** under **Refresh every**
5.  Click on the **Refresh** button
6.  A timestamp of the last refresh is displayed next to the **Refresh** button

**![](https://files.helpdocs.io/6m3dgskxth/articles/5wzpeh50f8/1609356294034/monitoring-screen-shot-5.png)**

> There is an interplay between the refresh and the resolution that is important to note. For example, with Last week, the resolution is 1 hour. However, if you select Real time or Recent with a frequency of a 30 second interval, because the resolution is 1 hour, the data will be refreshed at the top of the hour. Ex: 12 pm, 1 pm, 2 pm and so forth. If something happened at 12:15 pm, you will not see it until 1 pm when the data is refreshed.

### Aggregate Data

You can choose how you want to aggregate a rollup of data

- The options are percentiles, minimum, average, or maximum.

**![](https://files.helpdocs.io/6m3dgskxth/articles/5wzpeh50f8/1611889374157/monitoring-aggregate-options-screen-shot.png)**

### Graph Overlays

On top of the default metric graphic, you can draw additional overlay lines

- The options are **Statistics** and **Bands**
- Click on the **Alarm** overlay to see when Alarms were raised and resolved
- Alarm details are listed in the Annotations window
- Click on **Clear All** to remove an overlay

**![](https://files.helpdocs.io/6m3dgskxth/articles/5wzpeh50f8/1611889205010/monitoring-band-overlay.png)**

### 3, 2, 1 Column Display

- Charts can be displayed in 3, 2, or 1 columns

**![](https://files.helpdocs.io/6m3dgskxth/articles/5wzpeh50f8/1611888535438/monitoring-columns.png)**

### Ad Hoc Metrics

1.  Click on the magnifying glass in the upper right hand corner
2.  Enter the metric name or partial name
3.  Click on the metric and it will be added with an **X** next to the name
4.  Click on the **X** to delete the metric

**![](https://files.helpdocs.io/6m3dgskxth/articles/5wzpeh50f8/1611888430102/monitoring-ad-hoc.png)**

### Zoom In

1.  Highlight the area you want to zoom in on
2.  The graph will zoom in
3.  Repeat this to zoom in further
4.  Click **Reset zoom** to go back to the original display

**![](https://files.helpdocs.io/6m3dgskxth/articles/5wzpeh50f8/1611888331946/monitoring-zoom-1.png)**

**![](https://files.helpdocs.io/6m3dgskxth/articles/5wzpeh50f8/1611888342005/monitoring-zoom-2.png)**


MetricExplorer


Very often, when we are monitoring the health of a resource or troubleshooting, we need to correlate multiple pieces of data to get an accurate picture. A metric set allows you to collect data across a group of metrics in parallel. You can set up metric sets for host, container, and pod resources.

Here is how you configure metric sets:

#### Video Tutorial

<youtube id="sCBETgyZKgs" />

#### Step-by-Step Instructions

##### Create a Metric Set

1.  In **Resources**, select the resource you are setting up a metric set for
2.  At the bottom of **Metric sets**, click the **+** sign
3.  A **Create new metric set** form will be presented
4.  Provide a **Name** and **Description** for the new metric set
5.  From **Metric list**, select a metric and click on the → arrow to place it in the metric set
6.  You can rearrange the **Metric sequence** by dragging and dropping the metrics
7.  You can remove a metric by clicking on the **X**
8.  Click **Save**

##### Edit a Metric Set

1.  In **Metric sets**, select the metric set to edit
2.  To add a new metric, select it from **Metric list** and click on the → arrow to place it in the metric set
3.  To remove a metric, click on the **X** next to the metric name
4.  To update the existing metric set, click **Update**
5.  To save it as a new metric set, enter a new _Name_ and _Description_ and Click **Save as new**

##### Delete a Metric Set

1.  In **Metric sets**, select the metric set to delete
2.  At the bottom of **Metric sets**, click the **\-** sign
3.  Enter the metric set name (case sensitive) in the **Delete metric set** window
4.  Click **Delete**


Configuring Metric Sets


Resources in your environment have identifying information associated with them, known as tags. Tags indicate that a resource belongs to a particular category. We use tags to organize and filter resources into logical groups for monitoring, remediation...etc. In Shoreline, these logical groups are called dimensions. Up to 7 tags can be selected to create a dimension.

Once you have selected your dimensions, you can also specify the parent/child hierarchy of how the resources are displayed.

Here is how you configure dimensions:

#### Video Tutorial

<youtube id="Bt6ctpJ9QyE" />

#### Step-by-Step Instructions

##### Create a Dimension

1.  Under **Resources**, click on the resource you want to create a dimension for
2.  At the bottom of **Dimensions and Hierarchies**, click the **+** sign
3.  In the **Create new dimension or hierarchy** pane, select up to 7 dimensions under **Available tags**
4.  In the **Hierarchy** pane, you can drag and drop the tags to change the order
5.  Click **Create** and you will see the new dimension listed under **Dimensions and Hierarchies**

##### Edit a Dimension

1.  Click on the dimension under **Dimensions and Hierarchies**
2.  Add new dimensions (up to 7 total) or rearrange them
3.  Click **Update**

##### Delete a Dimension

1.  Select the dimension under **Dimensions and Hierarchies**
2.  Click the **\-** sign
3.  Confirm by clicking **Yes, remove**


Configuring Dimensions


Bots bind alarms and actions together and specify the action or set of actions to take when the alarm is raised.

Here is how you configure bots:

#### Video Tutorial

<youtube id="kevYKcUjGTU" />

#### Step-by-Step Instructions

##### Create a Standard Bot

1.  Click on **New Bot**
2.  Enter a **Name** and **Description**
3.  Leave the **Enable bot** box checked
4.  Select **Standard** for **Bot type**
5.  Select the alarm you want to create a bot for under **Alarm list**. If an alarm already has a bot associated with it, you will see that denoted to the right of the alarm name)
6.  Select action(s) that you want the bot to take from the **Action list** and move to the **Action sequence** panel
7.  Drag and drop the actions in the order you want them to be executed
8.  Click **Save bot**
9.  The bot name will now be listed next to **Associated bot** in the **Alarm detail** panel

##### Edit a Bot

1.  From the **Configured Bot list**, click on a bot
2.  Modify the bot by adding, deleting, or changing the action sequence
3.  Click **Save**

##### Delete/Disable a Bot

1.  To Disable or Delete, select the Bot by clicking on the checkbox next to the Bot name
2.  In the upper right hand corner of the dashboard, click on the dropdown next to **Select**
3.  Click **Disable** or **Delete**
4.  To delete, you will be prompted to enter the Bot name as a confirmation before deleting it


Configuring Bots


Alarms notify operators when a condition or threshold has been met. The ability to automate alarms and notify operators proactively is a key capability of DevOps.

Here is how you configure alarms:

#### Video Tutorial

<youtube id="c3JPXM9qQ6g" />

#### Step-by-Step Instructions

##### Create a Metric Alarm

1.  Click on **New Alarm**
2.  Enter a **Name** and **Description**
3.  Leave the **Enable alarm** box checked
4.  Select **Metric** for **Alarm based on**
5.  Select the **Resource Type** and apply a filter
6.  Select the **Metric** from the drop down
7.  Set the **Threshold** value - this can be a whole number or decimal
8.  Set the **Raise alarm when** and **Resolve alarm when** values - these are whole numbers
9.  Set the **Alarm description** and **Resolve description**
10. Optionally, set **Mute when** to disable the alarm under certain conditions
11. Click **Save**

##### Edit an Alarm

1.  From the **Configured alarm list**, click on an alarm
2.  Modify the alarm settings
3.  Click **Save**

##### Delete/Disable an Alarm

1.  To Disable or Delete, select the alarm by clicking on the checkbox next to the alarm name
2.  In the upper right hand corner of the dashboard, click on the dropdown next to **Select**
3.  Click **Disable** or **Delete**
4.  To delete, you will be prompted to enter the alarm name as a confirmation before deleting it


Configuring Alarms


Actions are shell commands and scripts that act upon the resource(s) to resolve the issue that is identified in an alarm.

Here is how you configure actions:

#### Video Tutorial

<youtube id="brZ1nvb1y3s" />

#### Step-by-Step Instructions

##### Create an Action

1.  Click on **New Action**
2.  Enter a **Name** and **Description**
3.  Leave the **Enable action** box checked
4.  Enter **Start**,  **Completion**, and **Error** titles and descriptions
5.  Click **Save**

##### Edit an Action

1.  From the **Configured action list**, click on an action
2.  Modify the action settings
3.  Click **Save**

##### Delete/Disable an Action

1.  To Disable or Delete, select the action by clicking on the checkbox next to the action name
2.  In the upper right hand corner of the dashboard, click on the dropdown next to **Select**
3.  Click **Disable** or **Delete**
4.  To delete, you will be prompted to enter the alarm name as a confirmation before deleting it


Configuring Actions


Alarms tell you when something has gone wrong and/or needs attention.  Alarms are defined on metrics, resources, and system state and are raised when a condition is met. The Alarms dashboard allows you to view alarm activity and details for a given period of time.

If you need to learn how to configure alarms or to refresh your knowledge, click here:

[Configuring Alarms](/v1/ui/configuring-alarms)

Here is how you use the Alarms Dashboard:

#### Video Tutorial

<youtube id="5UivmhobA18" />

#### Step-by-Step Instructions

##### Alarms Dashboard

When an Alarm is raised, detailed information about the alarm is logged and displayed on the dashboard in the Alarms module:

- **Start** - date and time
- **Duration** – how long the alarm was active or has been active
- **Status** - Raised or Resolved
- **Name** - you can sort and filter by Name
- **Description**
- **Resource** - Host, Pod, or Container
- **Resource name** - you can sort and filter by Resource name
- **Condition** - the threshold or value that raised or resolved the alarm

![](https://files.helpdocs.io/6m3dgskxth/articles/sr3vctqbfu/1612046433484/monitoring-alarms-1.png)

![](https://files.helpdocs.io/6m3dgskxth/articles/sr3vctqbfu/1612046803694/monitoring-alarms-2.png)

The dashboard shows the latest status of an alarm.  Let’s say an alarm was raised at 9:00 am and was resolved at 9:06 am.  The dashboard would show the alarm with **Status** as _Raised_ at 9:00 am.  Between 9:00 am and 9:06 am, the alarm would continue to show **Status** as _Raised_.  When it was resolved at 9:06 am, the **Status** would change to _Resolved_.

##### Monitor by Time Period

You can monitor alarms for a specific time period

1.  Click on **Period**
2.  The options are **Last 7 days**, the **Last 24 hours**, or the **Last 60 minutes**

![](https://files.helpdocs.io/6m3dgskxth/articles/sr3vctqbfu/1612049359915/monitoring-alarms-3.png)

##### Refreshing Data

You can select how the alarm data is refreshed

1.  **Refresh** mode is _Real time_ or _Recent_
2.  In _Recent_ you are getting metrics from the Shoreline back end
3.  In _Real time_, you are getting data from the Shoreline agent which is the most up-to-date to the last second possible
4.  In addition to the **Refresh** mode, you have to set the Refresh frequency under **Refresh Every**
5.  The options are every 1, 5, 10, or 30 seconds, or 1 minute.

![](https://files.helpdocs.io/6m3dgskxth/articles/sr3vctqbfu/1612054345465/monitoring-alarms-drill-down-refresh.png)

If you select the Manual mode, you have to refresh the data by clicking on the **Refresh** button. The timestamp of the last referesh is listed to the left.

##### Alarms Drill Down

From the dashboard, you can drill down on a specific alarm for details

1.  Click on a specific alarm **Name**
2.  The first pane shows alarm summary information as well as the impacted resource

![](https://files.helpdocs.io/6m3dgskxth/articles/sr3vctqbfu/1612053479389/monitoring-alarms-drill-down-1.png)

3.  The graph shows the activity before the alarm, when the alarm was raised and resolved, and the activity afterwards

![](https://files.helpdocs.io/6m3dgskxth/articles/sr3vctqbfu/1612053694700/monitoring-alarms-drill-down-2.png)

4.  The detailed log of events shows the alarm lifeycle from **Alarm Raised** to **Alarm Resolved**

![](https://files.helpdocs.io/6m3dgskxth/articles/sr3vctqbfu/1612053969071/monitoring-alarms-drill-down-3.png)

##### Zoom In

1.  Highlight the area you want to zoom in on
2.  The graph will zoom in
3.  Repeat this to zoom in further
4.  Click **Reset zoom** to go back to the original display

To return to the dashboard, click on **Go to Triggered Alarm list** in the upper left-hand corner.


Alarms Dashboard


Actions are shell commands and scripts that act upon the resource(s) to resolve the issue that is identified in an alarm. Bots bind alarms and actions together and specify the action or set of actions to take when an alarm is raised.

If you need to learn how to configure actions or bots or to refresh your knowledge, click here:

[Configuring Actions](/v1/ui/configuring-actions)

[Configuring Bots](/v1/ui/configuring-bots)

#### Video Tutorial

<youtube id="ewFUIDwzalQ" />

#### Step-by-Step Instructions

##### Actions Dashboard

When an Action or Bot is executed in response to an Alarm, information is displayed on the dashboard in the Alarms module.

For Actions:

- **Fired Timestamp**
- **Status** - Completed or Failed
- **Name** - you can sort and filter by Name
- **Description**
- **Resource type** - Host, Pod, or Container
- **Resource name** - you can sort and filter by Resource name
- **Execution Source** - the Bot name you can sort and filter on

![](https://files.helpdocs.io/6m3dgskxth/articles/zdcbup2qx8/1612055941413/monitoring-actions-1.png)

For Bots:

- **Name**
- **Alarm** - the condition that triggered the Bot to be executed
- **Actions**
- **Timestamp**
- **Status** - In Execution, Completed

![](https://files.helpdocs.io/6m3dgskxth/articles/zdcbup2qx8/1612055991985/monitoring-bots-1.png)

##### Monitor by Time Period

You can monitor actions and bots for a specific time period

1.  Click on **Period**
2.  The options are **Last 7 days**, the **Last 24 hours**, or the **Last 60 minutes**

![](https://files.helpdocs.io/6m3dgskxth/articles/zdcbup2qx8/1610073846610/actions-screen-shot-4.png)

##### Refreshing the Dashboard

You can select how the data is refreshed

1.  **Refresh** mode is _Real time_ or _Recent_
2.  In _Recent_ you are getting metrics from the Shoreline back end
3.  In _Real time_, you are getting data from the Shoreline agent which is the most up-to-date to the last second possible
4.  In addition to the **Refresh** mode, you have to set the Refresh frequency under **Refresh Every**
5.  The options are every 1, 5, 10, or 30 seconds, or 1 minute.

![](https://files.helpdocs.io/6m3dgskxth/articles/zdcbup2qx8/1610074119815/actions-screen-shot-5.png)

If you select the Manual mode, you have to refresh the data by clicking on the **Refresh** button. The timestamp of the last referesh is listed to the left.

##### Actions Drill Down

From the dashboard, you can drill down on a specific action for details

1.  Click on a specific action **Name**
2.  The first pane shows the action summary information
3.  The second pane shows the action output
4.  Click on the action **Name** to see the detailed commands or code

![](https://files.helpdocs.io/6m3dgskxth/articles/zdcbup2qx8/1612057680321/monitoring-actions-drill-down-1.png)

You can get details on the Action from the perspective of the alarm

1.  Click on the alarm **Name** and a new tab opens
2.  The first pane shows the alarm summary information

![](https://files.helpdocs.io/6m3dgskxth/articles/zdcbup2qx8/1612057997230/monitoring-actions-drill-down-2.png)

3.  The graph shows the activity before the alarm, when the alarm was raised and resolved, and the activity afterwards
4.  The detailed log of events shows the alarm lifeycle including the bot execution

![](https://files.helpdocs.io/6m3dgskxth/articles/zdcbup2qx8/1612058322109/monitoring-actions-drill-down-3.png)

You can also get details on the Action from the perspective of the bot

1.  Click on the bot **Name** and a new tab opens
2.  The first pane shows the alarm summary information
3.  The graph shows the activity before the alarm, when the alarm was raised and resolved, and the activity afterwards
4.  The detailed log of events shows the bot exection lifeycle

![](https://files.helpdocs.io/6m3dgskxth/articles/zdcbup2qx8/1612058651264/monitoring-actions-drill-down-4.png)

##### Zoom In

1.  Highlight the area you want to zoom in on
2.  The graph will zoom in
3.  Repeat this to zoom in further
4.  Click **Reset zoom** to go back to the original display

To return to the dashboard, click on **Go to Triggered Action list** or **Go to Triggered Bot list** in the upper left-hand corner.


Documentation

Actions

Bots

Configuration

UI

Explore the Op Language